Summary
-------

Update the organization of the documents in the JDK, in both the
source repositories and the generated docs.

Goals
-----

* Formally define an organization for the generated "docs" image, to include API
  specifications, "man pages" (which can be considered to be specifications for
  tools),  and other JDK specifications.
* Consolidate the current 20+ sets of documentation generated by the javadoc tool
  into a single collection of API specification for a JDK image.
* Define an organization for non-API specifications to be present in the
  source repositories, such that they can be updated as needed along with the
  source code, and can easily be included in the generated "docs" image.

Non-Goals
---------

* It is not a goal (it is an anti-goal) to change any of the processes or
procedures by which it is decided to update any specifications. 
This includes JCP specifications, such as the API specifications, and 
related standards.
* It is not a goal to include all specifications in this work. For example, 
JLS and JVMS are not included in this proposal.
* It is not a goal to support documentation that is not a specification.
* Although it is a goal to define an organization that can accommodate man
pages, it is not a goal to provide man pages for JDK 9.

Motivation
----------

Currently, a standard build of the JDK "docs" target runs the javadoc tool
over 20 times (22 on Linux, 25 on Solaris), to generate a corresponding number
of different sets of documentation, organized in no obviously discernible
manner. Even though javadoc documentation now provides a "Search" feature,
having multiple sets of documentation implies that you cannot easily search all
the provided documentation at once.

Here is a listing of the current collection of documentation sets; each entry
corresponds to a separate invocation of the javadoc tool. The first entry
(`api`) is the one that most people are familiar with: it is the Java SE
Platform Specification.

    api
    jdk/api/attach/spec
    jdk/api/dynalink
    jdk/api/javac/tree
    jdk/api/javadoc/doclet
    jdk/api/javadoc/old/doclet
    jdk/api/javadoc/old/taglet
    jdk/api/jconsole/spec
    jdk/api/jlink
    jdk/api/jpda/jdi
    jdk/api/jshell
    jdk/api/nashorn
    jre/api/accessibility/jaccess/spec
    jre/api/management/extension
    jre/api/net/httpserver/spec
    jre/api/net/socketoptions/spec
    jre/api/nio/sctp/spec
    jre/api/plugin/dom
    jre/api/plugin/jsobject
    jre/api/security/jaas/spec
    jre/api/security/jgss/spec
    jre/api/security/smartcardio/spec

Although there is a superficial overall organization, the varying depths, the
varying position of the `api` component, the inconsistent use of `spec`, all
make it difficult to determine what documentation sets exist, and what should be
the relative paths between related specifications.   Currently, the only "index"
into these pages is implicit in the so-called  "[brick wall](https://docs.oracle.com/javase/8/docs/)" picture.

It would be good to significantly reduce the number of different sets of
documentation, and to organize them in a well-defined manner, such that it is
reasonable to create links between and into the sets of documentation.

These various runs of the javadoc tool are currently controlled by logic in the
files `make/common/CORE_PKGS.gmk` and `make/common/NON_CORE_PKGS.gmk`.  Updating
these files is error-prone and sometimes overlooked. For example, there are
currently four supported packages that are not included in any public
documentation. It would be better to automatically derive the list of packages
to be documented from the list of packages exported by the modules being
documented. This would mean that any time a new package is listed as being
exported from a module, it would automatically included in any documentation
that includes the documentation for the module.

In addition, OpenJDK does not provide "man pages" for the tools it provides,
even though they can be considered the specification of the tools' command
lines.  It would be good to incorporate such specifications into the
repositories,  such that they can be updated alongside the corresponding tools
when necessary,  and corresponding pages generated and places in a well-defined
manner in the  overall documentation bundle.  The same can also be said for
various additional specifications, such as the JNI Specification, or the Javadoc
Tag Specification,  which currently have no well-defined home.

Finally, the current documentation bundle is effectively "all or nothing".  If
we organized any documentation in the repositories such that it can  be
associated with the module to which it applies, we would be able to build images
containing specific subsets of images, together with corresponding
documentation. For example, if we build images for the different
[Compact Profiles](https://docs.oracle.com/javase/8/docs/technotes/guides/compactprofiles/compactprofiles.html),
we could easily build and publish the corresponding documentation.

Description
-----------

### Consolidated API documentation

The consolidation of the generated API documentation requires changes to the
makefiles such that we identify the modules containing the API to be documented
and generate a bundle containing all those modules for which all of their
exported API should be documented.  If that does not cover all the API
documentation that is currently generated, we should define the organization of
the additional sets of documentation. For example, define that the various sets
exist as siblings within a single top-level "api" directory.

    docs/api/<doc-set>/

&lt;doc-set> should be a general name to describe the corresponding contents,
such as "jdk" or "java.se". 

### 'Man pages'

Source code within a JDK repository is typically organized as follows:

     src/<module-name>/{share,<os-name>}/{classes,native,...}

It is proposed to extend the last such component with a new variant:

     src/<module-name>/{share,<os-name>}/man

The `man` directory should contain the source for the man pages for tools in the
enclosing module. Such files should use Markdown syntax, and be named for the
corresponding tool. For example, the source for the man page for the _javac_
tool should be in `src/jdk.compiler/share/man/javac.md`.

The build system should generate corresponding files in a new top-level
directory in the generated `docs` directory. The generated files could either be
in HTML, or could be in "man" format for those systems that support that. For
example, the generated man page for the _javac_ tool could be in

    docs/man/javac.html
    docs/man/man1/javac.1

### Additional specifications

Some modules may have some additional associated specifications, that is not
otherwise included in API specifications or man pages. For example,
documentation  comments are generally written according to the "Javadoc Tag
Specification", which should be updated in conjunction with updates to the
javadoc tool. (It is currently available within the Javac Tool Guide at the
somewhat unmemorable URL
[https://docs.oracle.com/javase/8/docs/technotes/tools/unix/javadoc.html#CHDFCBAD])

Here are some additional specifications that may be considered for inclusion:

* [JavaBeans spec](http://www.oracle.com/technetwork/java/javase/documentation/spec-136004.html)
* [Input Methods Framework Specification](http://download.java.net/java/jdk9/docs/technotes/guides/imf/spec.html)
* [Jar File Specification](http://download.java.net/java/jdk9/docs/technotes/guides/jar/jar.html)
* [Java Native Interface Specification](http://download.java.net/java/jdk9/docs/technotes/guides/jni/spec/jniTOC.html)
* [Java Remote Method Invocation](http://download.java.net/java/jdk9/docs/platform/rmi/spec/rmiTOC.html)

Such specifications should be placed in another new module-specific directory:

     src/<module-name>/{share,<os-name>}/specs

Files in such a directory should be copied to a `specs` directory in the
generated `docs` directory, with the exception of Markdown (`.md`) files, which
should be translated into HTML files. If a specification consists of more than a
single file, it is recommended that all the files should be grouped into an
appropriate subdirectory.

### Index

The build should generate a simple minimal top-level `index.html` file that
links to each item in the overall generated documentation. This should be
enough for simple navigation in a basic "docs" build, but may be overwritten
as needed when providing a richer documentation bundle. 

### Formats

HTML files should be in either HTML 5 or HTML 4.01 format, and should pass
validators such as `tidy`, link checkers, and accessibility checkers. 
Over time, we should migrate all older HTML files to HTML 5, because of
the support for accessibility-related features.

Markdown files should be in one of the following formats:
[Markdown](http://daringfireball.net/projects/markdown); 
[CommonMark](http://commonmark.org);
or 
[Github Flavored Markdown](https://help.github.com/articles/github-flavored-markdown), 
which provides useful extensions for definition lists, tables, and syntax
highlighting of code blocks.

### Tools

It is proposed that we use the open source
[pandoc](http://pandoc.org/MANUAL.html) tool to convert Markdown files into HTML
or man page (groff) format. This would mean some new tool dependencies for the
build. `pandoc` supports all the Markdown variants listed in the previous
section.

### Summary

New source subdirectories:

    src/<module-name/{share,<os-name>}/
        man
        specs

New generated docs directory:

    docs/
        api/
            <doc-set-1>
            <doc-set-2>
            etc
        man/
            <HTML man pages>
            man1/
                <man pages in man format>
        specs/
            <HTML spec pages>
            <directories for multi-file specifications>