Summary
-------

Extend the Compiler Tree API to provide structured access to the content
of javadoc comments.


Goals
-----

Provide access to the syntactic elements of a javadoc comment.


Non-Goals
---------

It is a non-goal to check the HTML tags in a javadoc comment for semantic
well-formedness, that is, checking the HTML against a DTD or similar; 
however, it should be possible to use the API to provide such tools.


Motivation
----------

This API will enable a new generation of doc comment tools to be provided. Such
tools could either be written using the Compiler and Tree API, or could be
written as annotation processors. One tool that is long long overdue is the
updated equivalent of the old DocCheck doclet, to check simple rules and
guidelines for the contents of doc comments, and which has never been updated
for the language changes in Java 5 and later.

javadoc could also be rewritten to take advantage of the new structured doc
comment objects, and to be able to use the additional information such as
source positions in its error messages. The HTML parsing would also help make
javadoc be able to generate valid XHTML. Although the work in JDK 7 javadoc
makes it easy to generate XHTML for the sections generated by javadoc itself,
javadoc does not currently have the means to check or verify the use of XHTML
within the doc comments of the sources files it is processing.


Description
-----------

The problem(s) ...

In JDK 5, there was a single scanner, capable of reading doc comments as needed
by javadoc. In JDK 6, the code was refactored into two scanners, one that was
capable of reading doc comments, suitable for use by javadoc, and one that was
not, suitable for use by javac. That is, until we added the various public APIs
to javac, by which clients and annotation processors could get access to doc
comment if they so desired.

This means there are 3 types of clients for doc comments:

1. comments not required -- javac, when no annotation processors need to be run
2. comments definitely required -- javadoc
3. comments maybe required -- clients of javac public API, including annotation
   processors run by javac

The problem is that reading and maintaining the doc comment table is expensive,
and in category 3, we have to support the doc comment table, for the off chance
that clients might need it, even though most do not.

Another problem with the doc comment table is that it is very low-tech. It is
simply a map of tree node to string, where the string is the doc comment as
needed by javadoc, meaning that the beginning of each line (the white space and
typical '*') has been stripped away. This makes it very difficult indeed to
relate positions within the doc comment back to positions in the original
source file, which is why you don't see any traditional "emacs-style" error
messages coming from javadoc about "parameter name not found" or "exception not
declared to be thrown".

The same low-tech doc comment table is exposed to clients of the Tree API. For
any tree node, you can get the doc comment string. That's it, beyond that,
you're on your own.  The ideas(s) ...

First up is to upgrade the doc comment table stored in each compilation
unit. Replace

    Map<JCTree, String> docComments; 

by

    Map<JCTree, JCDocComment> docComments;

JCDocComment is a new object internal to javac, and provides lazy access to the
doc comment string. At a minimum it contains the starting position of the doc
comment in the source file: the position of the "/" character.

    interface JCDocComment {
        int getPosition();
        String getComment();
    }

This allows us to have possibly three different doc comment scanners, for the
three different kinds of client. For javac with no annotation processors, we
continue to use the standard Scanner and leave the docComments table empty, as
now. For javadoc, we continue to read the doc comments as now, except that now
we store them in JCDocComment objects. For javac when we don't know whether doc
comments are required or not, we simply store the starting position of the doc
comment. This saves us storing the text of all the doc comments when they are
not required. The price is that when we do need the comments, we have to go
back and recover the text of the comment from the source file. Different
strategies are possible. If any doc comment in a source file is required, we
could scan all of them. Note we don't have to scan the source text between the
comments because we have the starting position of the comment available, so we
can just skip the text between the comments. Or we could just read the comments
as needed, and rely on the content cache to save us having to read the source
file contents for each individual comment.

The next idea is a better, parsed, representation of doc comments.

A doc comment is comprised of

* an initial sentence
* the rest of the main description
* a list of tags each followed by a description

Each of these contains a sequence of fragments where each fragment can be one
of

* plain text, including characters from malformed fragments like '<',
  '>', '&', '{', etc.
* an HTML start-entity, which contains a name and a list of name-value pairs,
  such as '\<a href="Object.html">'
* an HTML end-entity, which contains a name, such as '\</a>'
* an HTML character entity, such as '\&amp;'
* a taglet, such as {@link Object}

Obviously, these can be modeled with a simple hierarchy of tree nodes, so I
suggest a new package, com.sun.source.doccomments to contain the interfaces for
these tree nodes. It would best be a separate hierarchy from the existing
com.sun.source.tree.Tree, so I suggest a new common super-interface
com.sun.source.doccomments.DTree. We can then extend the utility methods in
com.sun.source.util, to provide access to the parsed doc comment for any tree
node, and to provide source position info for any DTree node.

Note, the HTML start-entity and end-entity are handled as separate items to
avoid getting into issues of parsing HTML and knowing which tags require a
closing tag and which not. That layer can be built on top of this abstraction
by those applications that need it.

Parsing these comments will not be cheap; nothing involving lexing and parsing
ever is. And so this is another reason to provide and use the lazy access to
doc comments via the JCDocComment table described earlier. Except now, there is
a more interesting method on it as well, to get the DTree for a comment --
which is another reason not to bother to keep the simple string that is
currently provided.


Testing
-------

langtools regression tests will be written to exercise the new API.
One specific test will be to read and process all the JDK API comments.

There are no special platform or hardware requirements.


Dependences
-----------

This work has no dependences on other JEPs. 

It is expected that other JEPs will depend on this one.


Impact
------

  - Other JDK components: javadoc
  - Compatibility: minimal
  - Internationalization: minimal
  - Localization: minimal