Summary
-------

Reduce the time required to build the JDK and enable incremental builds by
modifying the Java compiler to run on all available cores in a single
persistent process, track package and class dependences between builds,
automatically generate header files for native methods, and clean up class and
header files that are no longer needed.


Goals
-----

The top level goals are:

  1. Increase build speed by having javac use all cores and reuse javac in a
     server process.
  2. Simplify work for developers by having javac build incrementally.

This project is part of a larger effort to improve the build infrastructure of
the JDK.

The improvements to javac will be internal and not available through the public
api of the javac launcher. Instead an internal wrapper program called
smart-javac (or sjavac for short) will house the new functionality.
Eventually, when the javac wrapper features have stabilized, they can be
proposed in a future JEP to be moved to the public api of javac.  This would
allow all Java developers to take advantage of the improvements.


Non-Goals
---------

This project only concerns the changes needed in javac and the new wrapper. It
does not cover the changes needed in the JDK Makefiles to take advantage of
these changes; those are described in
[JEP 138: Autoconf-Based Build System](JDK-8046128).

This project will not concern changes needed in javac to speed up Javadoc
generation.


Success Metrics
---------------

All cores should be used when compiling Java sources and there should be a
speedup in build performance when using multiple cores.

The workload distributed on the different cores will not be perfectly balanced
and it is known that the same work will be recomputed several times.  But the
changes supported by this JEP will enable us to successively improve the
sharing of work between the cores within javac.

An incremental build should recompile only the changed packages and their
dependencies.

After an increment build has been done, where classes or native methods were
removed, the output directories should be clean; _i.e._, no classes or C-header
files corresponding to the removed sources should remain.


Motivation
----------

Building the complete OpenJDK is unnecessarily slow. This puts an extra burden
on developers and build systems. As a result, developers check out and build
just a part of the source code, since the product as a whole takes too long to
build.


Description
-----------

The internal smart-javac wrapper will probably be invoked like this:

    $ java -jar sjavac.jar -classpath ... -sourcepath ... -pkg '*' \
           -j all -h headerdir -d outputdir

This will compile all source files found in the sourcepath with package names
matching anything (`'*'`) using all (`-j`) cores.  A database file will be
created in the (`-d`) outputdir called `.javac_state` which will contain all
information necessary to do a fast incremental compile, with proper cleanup of
disappearing classes and C-headers as well as correct dependency tracking.

The smart-javac wrapper implements multi core support by creating a
JavaCompiler instance for each core. The source code to be compiled is then
split into packages and randomly distributed to the JavaCompilers.  If a
randomly selected package has dependencies, then these dependencies will be
compiled automatically, but not written to disk, _i.e._, `-Ximplicit:none`. If
an implicitly compiled dependency is later requested as part of a randomly
selected package then the implicit work is not wasted; instead the
already-compiled dependency is written to disk.

Since the initial compile does not know which packages a package depends upon,
a random distribution of work is the best we can do. Therefore
`java.lang.Object` will be recompiled as many times as there are cores, but
only one of the JavaCompilers will be responsible for writing
`java.lang.Object` to disk. Enough packages are independent of each other to
make this a workable strategy for making use of multiple cores.

When we improve javac in the future (not part of this JEP), more and more work
will be shared between the JavaCompilers and eventually we will reach the state
where `java.lang.Object` is only compiled once.

javah will be automatically run on any class that contains native methods and
the generated C-headers will be put in the (`-h`) headerdir. A new annotation
`@ForceNativeHeader` is used for classes that have final static primitives that
need to be exported to JNI, but no native methods.

To avoid restarting javac and losing optimizations done by the JVM, the
smart-javac wrapper supports a `-server` option. This option will spawn a
background javac server so that each subsequent smart-javac wrapper invocation
that refers to the same portfile will reuse the same server.

The `-server` arguments are:

  - `portfile` = where to store the TCP port used
  - `logfile` = where to store the output from javac, defaults to
    portfile+".logfile"
  - `stdouterrfile` = where to store output from the server, defaults to
    portfile+".stdouterr"
  - `javac` = the path to the javac to launch by the server, with spaces and
    commas replaced by `%20` and `%2C`.

Example:

    -server:portfile=/tmp/jdk.port,javac=/usr/local/bin/java%20\
    -jar%20/tmp/openjdk/langtools/dist/lib/bootstrap/sjavac.jar

Since javac cannot currently share state between concurrent compilations, each
additional core will consume roughly as much memory as a single invocation of
javac. Improved sharing between the cores will be successively introduced after
this JEP is completed. The `-j` option can be used to limit the number of cores
and thus the memory usage.

The javac server stays in memory after all compilations have finished.  The
server will automatically shut down and free memory and other resources after
30 seconds of inactivity.

The server runs as the same user as the normal javac compiler, and thus has the
same privileges and possibility to write to the build output directory. Unlike
the normal javac compiler, a compilation can be trigged by connecting to the
server through a TCP port. Only the command line, and not the source code, is
sent over TCP.

A potential security risk is that an attacker could add the compilation of some
malicious piece of code, which would appear in the output directory. To
alleviate this risk, we will use the several measures:

  - Open a new TCP port each time; the port number is stored in the portfile.
  - Only allow connections from localhost.
  - Require a unique cookie to be presented to the server before any
    compilation will occur.

The cookie is a 64 bit random integer stored in the portfile. The portfile has
typical temporary file permissions, _i.e._, only the owner is allowed to read
from it or write to it.


Alternatives
------------

Instead of making javac properly parallel, we could start several
single-threaded compilations of different and independent java packages in
parallel. This would not require any changes to javac, but it would be much
harder to get the Makefiles correct, and it would not give as much speed
improvement.


Testing
-------

The build infrastructure project will test that the output of the old build
system and the new build system is identical.  This will make sure that the
smart-javac wrapper generates identical output for the same source files.

None of the new options are going to be public, so no tests need to be added to
the javac test suite, but more-specific tests for the smart-javac wrapper will
be added to the langtools test directory.


Risks and Assumptions
---------------------

Resulting product is incorrect

  - Risk: Javac changes causes incorrect bits to be build
  - Mitigation plan: Test resulting build properly

The old build makefiles will be available concurrently with the new makefiles
to simplify comparing of the old and new bits.


Dependences
-----------

This JEP does not depend on any other changes. It forms the basis for
[JEP 138: Autoconf-Based Build System](JDK-8046128), which will use this new
feature in javac to speed up JDK builds.  The new makefiles can use
an unmodified javac, but they will not achieve the desired speedup
and incremental build support unless this JEP is completed.


Impact
------

  - Compatibility: Low impact. We will not add new arguments to javac.
  - Security: Low impact. In the description section, there is a discussion
    about the security aspects of opening up a server for compilation jobs.
  - I18n/L10n: The new smart-javac wrapper features will most likely result in
    a few new messages; we will not fully translate these since smart-javac is
    not yet part of any public api.
  - Testing: Apart from the build itself, separate test cases for the different
    smart-javac options should be written.