Summary
-------

Extend the JAR file format to allow multiple, Java-release-specific
versions of class files to coexist in a single archive.


Goals
-----

  1. Enhance the Java Archive Tool (`jar`) so that it can create
     multi-release JAR files.

  2. Implement multi-release JAR files in the JRE, including support
     in the standard class loaders and `JarFile` API.

  3. Enhance other critical tools (e.g., `javac`, `javap`, `jdeps`,
     etc.) to interpret multi-release JAR files.

  4. Support multi-release **modular** JAR files for goals 1 to 3.
  
  5. Preserve performance: The performance of tools and components that
     use multi-release JAR files must not be significantly impacted.  In
     particular, performance when accessing ordinary (i.e., not
     multi-release) JAR files must not be degraded.


Motivation
----------

Third party libraries and frameworks typically support a range of Java 
platform versions, generally going several versions back.  As a 
consequence they often do not take advantage of language or API features 
available in newer releases since it is difficult to express conditional 
platform dependencies, which generally involves reflection, or to 
distribute different library artifacts for different platform versions.

This creates a disincentive for libraries and frameworks to use new 
features, that in turn creates a disincentive for users to upgrade to 
new JDK versions---a vicious circle that impedes adoption, to everyone's 
detriment.

Some libraries and frameworks, furthermore, use internal APIs of the JDK 
that will be made inaccessible in Java 9 when module boundaries are 
strictly enforced.  This also creates a disincentive to support new 
platform versions when there are public, supported API replacements for 
such internal APIs.


Description
-----------

A JAR file has a content root, which contains classes and resources, as 
well as a `META-INF` directory which contains metadata about the JAR.
By adding some versioning metadata to specific groups of files the JAR 
format can encode, in a compatible way, multiple versions of a library 
for different target Java
platform releases.

A multi-release JAR ("MRJAR") will contain the main attribute:

    Multi-Release: true

declared in the main section of the JAR `MANIFEST.MF`. The attribute 
name is also declared as a constant `java.util.jar.Attributes.MULTI_RELEASE`.
Like other main attributes the name declared in the `MANIFEST.MF` is 
case insensitive.  The value is also case-insensitive, but there must be 
no preceding or trailing white space (such a restriction helps ensure
the performance goal is met).

A multi-release JAR ("MRJAR") will contain additional directories for 
classes and resources specific to particular Java platform releases.  A 
JAR for a typical library might look like this:

    jar root
      - A.class
      - B.class
      - C.class
      - D.class

Suppose there are alternate versions of A and B that can take advantage 
of Java 9 features.  We can bundle them into a single JAR as follows:

    jar root
      - A.class
      - B.class
      - C.class
      - D.class
      - META-INF
         - versions
            - 9
               - A.class
               - B.class

In a JDK that does not support MRJARs, only the classes and resources in 
the root directory will be visible, and the two packagings will be
indistinguishable.  In a JDK that does support MRJARs, the directories
corresponding to any later Java platform release would be ignored; 
it would search for classes and resources first in the Java 
platform-specific directory corresponding to the currently-running major 
Java platform release version, then search those for lower versions, and 
finally the JAR root.  On a Java 9 JDK, it would be as if there were a 
JAR-specific class path containing first the version 9 files, and then 
the JAR root; on a Java 8 JDK, this class path would contain only the 
JAR root.

Suppose later on in the future Java 10 is released and A is updated to 
take advantage of Java 10 features. The MRJAR may then look like this:

    jar root
      - A.class
      - B.class
      - C.class
      - D.class
      - META-INF
         - versions
            - 9
               - A.class
               - B.class
            - 10
               - A.class

By this scheme, it is possible for versions of a class designed for a 
later Java platform release to override the version of that same class 
designed for an earlier Java platform release.  In the example above, 
when running on an MRJAR-aware Java 9 JDK, it would see the 9-specific 
versions of A and B and the general versions of C and D; on a future 
MRJAR-aware Java 10 JDK, it would see the 10-specific version of A and 
the 9-specific version of B; on older or non-MRJAR-aware JDKs it would 
only see the root versions of all.

JAR metadata, such as that found in the `MANIFEST.MF` file and the
`META-INF/services` directory, need not be versioned.  An MRJAR is 
essentially one unit of release, so it has just one release version 
(which is no different from a normal JAR, distributed say via Maven 
Central), even though internally it contains multiple versions of a 
library implementation for use on different Java platform releases.
Every version of the library should offer the same API; investigation is 
required to determine whether this should be strict backwards 
compatibility where the API is exactly the same (byte code signature 
equality), or whether this can be relaxed to some degree without
necessarily enabling the introduction of new enhancements that would 
blur the notion of one unit of release.  This may imply, at a minimum, 
that a public class present in a release-specific directory should also 
be present in the root, though it need not be present in an earlier 
release directory.  The run-time system will not verify this property, 
but tooling can and should detect such API compatibility issues, and a 
library method may also be provided to perform such varification (for 
example on `java.util.jar.JarFile`).

Ultimately, this mechanism enables library and framework developers to 
decouple the use of APIs in a specific Java platform release version 
from the requirement that all their users migrate to that version.
Library and framework maintainers can gradually migrate to and support
new features while still carrying around support for the old features, 
breaking the chicken-and-egg cycle so that a library can be 
"Java 9-ready" without actually requiring Java 9.

### Details

The following components of the JDK will be modified in order to support 
multi-release JAR files.

  - The JAR-based `URLClassLoader` must read selected versions of class
    files as indicated by the running Java platform version.  The
    module-based class loader introduced with Project Jigsaw will 
    require similar modifications.

  - The protocol handler for the `jar` URL scheme and
    `java.util.jar.JarFile` class must select the appropriate version of 
    a class from a multi-release JAR.

  - The Java compiler (`javac`), via the underlying `JavacFileManager`
    and ` ZipFileSystem` APIs, must read selected versions of class 
    files as specified by the `-target` and `-release` command-line 
    options.  The tools `javah`, `schemagen`, and `wsgen` will leverage 
    the underlying changes to `JavacFileManager` and `ZipFileSystem`.

  - The Java Archive tool (`jar`) will be enhanced so that it can
    create multi-release JAR files.

  - The JAR packing tool (`pack200`/`unpack200`) must be updated (see
    [JDK-8066272](https://bugs.openjdk.java.net/browse/JDK-8066272)).

  - The `javap` tool must be updated to enable selection of versioned
    class files.

  - The `jdeps` tool will require modifications to display version
    information and follow version specific class file dependencies.

  - The JAR specification must be revised to describe the multi-release
    JAR file format and any related changes (e.g., possible additions to
    the manifest).

### Compatibility

By default the behaviour of `java.util.jar.JarFile` and the `jar` scheme 
protocol handlers will remain the same.  It is necessary to opt-in to 
construct a `JarFile` pointing to a MRJAR for version selection of 
entries.  Likewise it is necessary to opt-in for `jar` URLs (see next 
section for details).

`JarFile` instances created by the runtime for class loading will opt-in
and create instances that are configured to select entries according to 
the version of the running Java platform.  Such as `JarFile` instance
is referred to as being runtime versioned.

### Class loader resources

A resource URL, produced by a class loader, identifying a resource in a
MRJAR will refer directly to a versioned entry (if present).  For 
example for a versioned resource, `foo/baz/resource.txt`:

    URL r = loader.getResource("foo/baz/resource.txt"); 

the URL ���r��� may be:

    jar:file:/mrjar.jar!/META-INF/versions/9/foo/baz/resource.txt
    
rather than:

    jar:file:/mrjar.jar!/foo/baz/resource.txt

This approach is considered the least disruptive option. Changing the 
structure of resources URLs is not without risk (e.g. a new scheme or an 
appended fragment).  Legacy code may process URL characters directly, 
rather than parsing the URL and correctly extracting the components.
While such URL process is incorrect it was considered preferable to not 
breaking such code.

### Modular multi-release JAR files

A modular multi-release JAR file is a multi-release JAR file that has
a module descriptor, `module-info.class`, in the root at the top, just
like a modular JAR file (see the [Packaging: Modular JAR][JEP-261-modular-jar]
section of JEP 261).  In addition modular descriptors may be present
in versioned areas.  Such versioned descriptors must be identical to the 
root module descriptor, with two exceptions:

  - A versioned descriptor can have different non-`transitive` `requires`
    clauses of `java.*` and `jdk.*` modules; and

  - A versioned descriptor can have different `uses` clauses, even of
    service types defined outside of `java.*` and `jdk.*` modules.

[JEP-261-modular-jar]: http://openjdk.java.net/jeps/261#Packaging:-Modular-JAR-files

The reasoning here is that these are implementation details rather than
parts of a module's API surface, and that one may well want to change
them as the JDK itself evolves.  Changes to non-public `requires` of 
non-JDK modules are not allowed.  If that is necessary then new version
of the module is required (at least increasing it's version number) and 
this is a different kind of compatibility problem, and one that's beyond 
the scope of MRJARs.

A multi-release modular need not have a module descriptor at the located 
root.  In this respect a module descriptor would be treated no differently to 
any other class or resource file.  This can ensure that, for example, only 
Java 8 versioned classes are present in the root area while Java 9 versioned 
classes (including the module descriptor) are present in the 9 versioned area.

### Classpath and modulepath

A modular JAR can be constructed such that it works correctly on the
classpath of a Java 8 runtime, the classpath of a Java 9 runtime, or
the modulepath of a Java 9 runtime.  The situation is the same for a
modular multi-release JAR file (which in addition to the 
`module-info.class` other classes may be compiled for the Java 9 
platform).

If a module descriptor does not declare some packages as exported, and
therefore public classes in those packages are private to the module,
then when the corresponding JAR file is placed on the module path the 
classes will not be accessible.  However, if the JAR file is placed on
the classpath then those classes will be accessible.  This is an
unfortunate consequence of supporting the classpath and modulepath.

As a consequence the public API for multi-release JAR file may be 
different when placed on the classpath compared to when placed on the
module path.  Ordinarily the jar tool when constructing a multi-release 
JAR file will, on a best effort basis, fail if any observed differences 
in the public API are detected.  However, when constructing a modular 
multi-release JAR file it is proposed the jar tool output a warning if
public API differences are a result of module private classes being 
accessible when the JAR file is placed on the classpath.

### Multi-release jars and the boot loader

Multi-release JARs are not supported by the boot loader (for example, when a
multi-release JAR file is declared with the -Xbootclasspath/a option).  Such 
support would complicate the boot loader implementation for what is 
considered a rare use-case.

Alternatives
------------

A common approach is to use a static reflective check to determine if an 
API feature is present or not and accordingly select an appropriate 
class that respectively depends on that feature or not.  The reflective 
cost is incurred at class initialization and not every time the 
dependent feature is used.  A Java platform release is selected for 
compilation with the source and target flags set to a lower release to 
generate class files compatible with that lower release.  This approach 
is often augmented with tools such as [Animal Sniffer][animal-sniffer] 
to check for API incompatibilities, where in addition to enforcing API 
compatibility code can be annotated to state whether it depends on a 
later Java platform release.  There are a number of limitations with 
this approach:

  1. The reflective checks need to be carefully maintained.

  2. It is not possible to utilize newer language features.

  3. If a platform release API feature is removed (perhaps an internal 
     API) then dependent code will fail to compile.

[animal-sniffer]: https://github.com/mojohaus/animal-sniffer

"Fat" class files were considered, where a class may have one or more 
methods targeted to different Java platform versions.  This was deemed 
too complicated in terms of the language and runtime features required 
to support such method declarations and dynamic selection.

Method handles (`invokedynamic`) cannot be used because of the need to 
maintain binary compatibility.


Risks and Assumptions
---------------------

It is anticipated that the production of MRJARs is primarily compatible 
with existing popular build tools and therefore IDEs that support such 
tools, but the developer experience could be improved with enhancements.

The source layout and building of an MRJAR file can be supported by 
Maven using a multi-module project.  For example, see [this][mrjar-maven] 
example Maven project that can produce a, currently rudimentary, MRJAR 
file.  There would be a sub-project for the root and specific Java 
platform releases, and a sub-project to assemble the aforementioned 
sub-projects into an MVJAR.  The assembly process could be enhanced, 
perhaps using a specific Maven plugin, leveraging the same features as 
the `jar` tool to enforce backwards compatibility.

[mrjar-maven]: https://github.com/hboutemy/maven-jep238

The design and implementation of the runtime processing of MRJARs 
currently assumes that a runtime uses the URL class loader or a custom 
class loader leverages `JarFile` to obtain platform-specific class 
files.  Runtimes whose class loaders use `ZipFile` to load classes will 
not be MRJAR aware.  Popular application frameworks and tools, such as 
Jetty, Tomcat, and Maven, etc., need to be checked for compatibility.


Dependences
-----------

The [enhanced JAR-file format][mjar] under consideration for the
[Java Platform Module System][jpms] will need to take multi-release
JAR metadata into account.

[mjar]: http://openjdk.java.net/projects/jigsaw/spec/sotms/#module-artifacts
[jpms]: http://openjdk.java.net/projects/jigsaw/spec/

[JEP 247 (Compile for Older Platform Versions)][jep247], which supports
compiling against older versions of the platform libraries, may aid 
build tools in the production of multi-release JAR files.

[jep247]: http://openjdk.java.net/jeps/247