Summary
-------

Introduce _nests_, an access-control context that aligns with the existing notion of nested types in the Java programming language. Nests allow classes that are logically part of the same code entity, but which are compiled to distinct class files, to access each other's private members without the need for compilers to insert accessibility-broadening bridge methods. 

Non-Goals
---------

This JEP is not concerned with large scales of access control, such as modules.  

Motivation
----------

Many JVM languages support multiple classes in a single source file
(such as Java's nested classes), or translate non-class source
artifacts to class files.  From a user perspective, however, these are
generally considered to be all in "the same class", and therefore
users expect them to share a common access control regime.  To
preserve these expectations, compilers frequently have to broaden
the access of `private` members to `package`, through the
addition of access bridges: an invocation of a private member is 
compiled into an invocation of a compiler-generated package-private method in the 
target class, which in turn accesses the intended private member.
These bridges subvert encapsulation, slightly increase the size of a deployed application,
and can confuse users and tools.  A formal notion of a group of
class files forming a _nest_, where _nest mates_ share a common access
control mechanism, allows the desired result to be directly achieved in a
simpler, more secure, more transparent manner.

The notion of a common access control context arises in other places
as well, such as the host class mechanism in
`Unsafe.defineAnonymousClass()`, where a dynamically loaded class can
use the access control context of a host.  A formal notion of nest
membership would put this mechanism on firmer ground (but actually
providing a supported replacement for `defineAnonymousClass()` would
be a separate effort.)


Description
-----------

The Java Language Specification allows classes and interfaces to be
nested within each other.  Within the scope of a top-level declaration
(JLS 7.6), any number of types can appear nested. These nested types
have unrestricted access to each other (JLS 6.6.1), including to
private fields, methods, and constructors.  We can describe a
top-level type, plus all types nested within it, as forming a _nest_,
and two members of a nest are described as _nestmates_.

The private access is complete (undifferentiated, flat) within the whole declaration of the
containing top-level type.  (One can think of this as a top-level type
defining a sort of "mini-package", within which extra access is
granted, even beyond that provided to other members of the same Java
package.)

Today private access between nestmates is not permitted by the JVM access rules. 
To provide the permitted access a Java source code compiler has to introduce a level
of indirection. For example, an invocation of a private member is compiled into an invocation 
of a compiler-generated package-private, bridging method in the target class, which in turn invokes 
the intended private method. These access bridges are generated only as needed to satisfy
the member accesses requested within the nest.

A further consequence of the lack of JVM support for private access within a nest, is that core reflection
also denies access. A reflective method invocation (using `java.lang.reflect.Method.invoke`) 
from one nestmate to another throws `IllegalAccessError` (unless access control has been disabled). This is surprising
given that reflective invocations should behave the same as source level invocations.
Similarly, the `MethodHandle` API rejects a direct "lookup" of a private nestmate method, but provides special support through `Lookup.in` to allow
the source level invocation semantics to be expressed.

By codifying the notion of nestmates and the associated access rules
within the JVM, we simplify the job of Java source code compilers, tighten the existing
access checks and remove surprising behaviour from the core reflection and `MethodHandle` APIs.
We also allow for future enhancements to take advantage of the "nest" notion. For example: 

- In [generic specialization](http://cr.openjdk.java.net/~briangoetz/valhalla/specialization.html), each specialized type could be created as a nestmate of the generic type.  
- A safe and supported replacement for the `Unsafe.defineAnonymousClass()` API could create the new class as a
nestmate of an existing class.
- The concept of "sealed classes" could be effected by only allowing subclasses that are nestmates.
- Truly private nested types could be effected (presently private nested types are defined with package-access).

### Nest Class File Attributes

The existing classfile format defines the `InnerClasses` and `EnclosingMethod` attributes  (JVMS 4.7.6 and 4.7.7) 
to allow Java source code compilers, like `javac`, to reify the source level nesting relationship. 
Each nested type is compiled to its own class file, with different class files "linked" by
the value of these attributes. While these attributes are enough for the JVM to
determine nestmate-ness, they are not directly suitable for access control and are inherently tied to
a single Java language concept.

To allow for a broader, more general, notion of nestmates beyond simply Java language nested types, and for the
sake of efficient access control checking, it is proposed to modify the class file format to
define two new attributes. One nest member (typically the top-level class) is designated as the _nest host_, and contains an attribute (`NestMembers`) to
identify the other statically known nest members. Each of the other nest members has an attribute (`NestHost`) to identify
its nest host.

### JVM Access Control for Nestmates

We will adjust the JVM's access rules by adding something like the following clause to JVMS 5.4.4:

> A field or method _R_ is accessible to a class or interface
> _D_ if and only if any of the following conditions are true:
> <ul>
>   <li> ... </li>
>   <li> <it>R</it> is private and is declared in a different class or
>   interface <it>C</it>, and <it>C</it> and <it>D</it>, are nestmates. </li>
> </ul>

For types _C_ and _D_ to be nestmates they must have the same
nest host.  A type _C_ claims to be a member of the nest hosted by _D_, if it
lists _D_ in its `NestHost` attribute. The membership is
validated if _D_ also lists _C_ in its `NestMembers` attribute. _D_ is implicitly a member of the nest that it hosts.

A class with no `NestHost` or `NestMembers` attribute, implicitly forms a nest with itself as the nest host, and sole nest member.

The loosened access rules would affect access checks during the following activities:

  * Resolving fields and methods (JVMS 5.4.3.2, etc.)
  * Resolving method handle constants (JVMS 5.4.3.5)
  * Resolving call site specifiers (JVMS 5.4.3.6)
  * Checking Java language access by instances of `java.lang.reflect.AccessibleObject`
  * Checking access during queries to `java.lang.invoke.MethodHandles.Lookup`

With the change to the access rules, and with suitable adjustments to
byte code rules, we can allow simplified rules for generating invocation bytecodes:

 - `invokespecial` for private nestmate constructors,
 - `invokevirtual` for private non-interface, nestmate instance methods,
 - `invokeinterface` for private interface, nestmate instance methods; and
 - `invokestatic` for private nestmate, static methods

This relaxes the existing constraint that private interface methods must be invoked using `invokespecial` (JVMS 6.5)
and more generally allows `invokevirtual` to be used for private method invocation, rather than adding to the complex usage rules surrounding `invokespecial`.
Similar changes can be made to the semantics of `MethodHandle` invocation (which mirrors the invocation byte code constraints).

### Nest Membership Validation

Nest membership must be validated before an access check relying on nestmate access can proceed. This may happen as late as the time of access of the member, or
as early as verification time of the class, or somewhere in-between, such as JIT compilation of a method.
Nest membership validation requires loading of the nest host class if it has not already been loaded. To avoid potentially unnecessary class loading,
nest membership validation should be performed as late as possible i.e. at the time of the access check. This mitigates the impact of the incompatibility introduced by requiring
that the nest host class be present if nestmate access is relied upon.

To preserve integrity of the nest it is proposed that, at least initially, it is prohibited to modify the nest classfile attributes using any form of class transformation or class redefinition.

### Nestmate Reflection API

As we are introducing new classfile attributes it is customary to provide a means to inspect/query those attributes using core reflection.
This is currently envisaged as three methods in `java.lang.Class`: `getNestHost`, `getNestMembers`, and `isNestmateOf`.

### Affected Specifications and APIs

The proposed changes, while conceptually simple, affect all specifications and API's that explicitly, or implicitly, involve access control, or relate to method invocation modes. These include:

- The Java Virtual Machine Specification
  - Classfile attribute changes
  - Access control rule changes
  - Invocation bytecode rule changes
- Core reflection
  - `Method` invocation rules
  - `Field` access rules
- `MethodHandle` lookup rules
- Class transformation/re-definition: JVM TI and `java.lang.instrument` API, JDWP and JDI (`com.sun.jdi.VirtualMachine`)
  - Prohibit modification of the nest related classfile attributes
- Pack200 Specification
  - Recognize the new classfile attributes

### Impact on Java Source Code Compilers

The proposed changes simplify the rules for mapping Java source constructs to class files, and so have a number of affects on a Java source code compiler that chooses to utilize them:

- Proper generation of the nest related classfile attributes
- Elision of the previously needed access bridge methods and generation of direct member access instructions for private nestmate members
- Issuing of the correct/appropriate invocation bytecodes
- The ability to change other synthetic methods to be private rather than package-private (or even eliminate them, or replace them with shared-but-private method handle constants)

The `javac` compiler will be updated to fully utilize nestmates when generating the latest version classfiles. (Older versions will be generated as they are today using access bridges etc.)

### Impact on Other Tools

Any tool that operates on classfiles, or which generates or processes bytecodes is potentially impacted by these changes. At a minimum such tools must tolerate the presence of the new classfile attributes and allow for the change in bytecode rules. For example:

- The `javap` classfile inspection tool,
- The Pack200 implementation, and
- The [ASM bytecode manipulation framework](http://asm.ow2.org), which is also used internally in the JDK.

### Open Issues

The extra complexity of access checking is something that has to be examined. In particular issues surrounding the resolution of the nest host class and the errors that can arise. We have already encountered and addressed a problem where a compiler thread needed to load a nest host class - which is not permitted in the compiler thread. We need to ensure the implementation is able to handle these conditions, and ensure that the specification is not impacted by their introduction.

Alternatives
------------

We can continue generating bridge methods in the Java compiler, as
needed.  This is a hard process to predict.  For example, Project
Lambda had difficulty resolving method handle constants in the
presence of inner classes, leading to a new type of bridge method.
Because compiler-generated bridge methods are tricky and unpredictable, they
are also buggy and hard to analyze by various tools, including
decompilers and debuggers.

The initial proposal considered using the existing `InnerClasses` and
`EnclosingMethod` attributes to establish nestmate-ship. But
introducing specific nestmate related attributes both makes nestmates
more general than only relating to language-level nested types, and
permits a more efficient implementation. Additionally, if we had opted for 
eager nest membership validation checks it would have changed the semantics
of the existing attributes and that would have been a compatibility issue. 
While the `javac` compiler will likely keep the "inner classes" and "nest members" 
attributes aligned, this is a compiler choice, and the JVM will treat them completely independently.

There was discussion about how best to express the nesting relationship 
through the classfile attributes before settling on the current approach. 
One suggestion, for a de-centralised approach, was for each nest to be identified
by a UUID. That discussion [concluded as follows](http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2016-February/000101.html):

> There are two parts to such a proposall:
> 
>  1. New naming convention for nests, based on UUIDs.  This is a new concept in the JVM, and would require new infrastructure to manage (generate, transcode, verify, reflect, debug). That means new bugs and new attack surfaces.  In the absence of a decisive benefit, it's better to reuse existing name spaces, and (in particular) the JVM's type name dictionary.
> 
>  2. Unidirectional links.  The UUID, being a pure identity with no content, does not contain a list of its nest members.  The nest members point to the nest (via the UUID).  Any class can inject itself into a nest (in the same package) simply by mentioning the appropriate UUID.  Unidirectional linkage means that there is no way to enumerate a nest.  This complicates  some optimizations (based on sealed types).  Security and seal-ability of nests is reduced to that of packages.  PRIVATE becomes just an alias for default-scope access control.
>
> Sorry, but neither part of this is appealing to me, compared with the current proposal.

Testing
-------

We will need an extensive set of JVM tests to verify the new access rules and adjustments to the byte code semantics to support nestmates.

Similarly we will need additional tests for core reflection, method handles, var-handles, and external access API's like JDWP, JVM TI and JNI.

Since no language changes are proposed here, no new language compliance tests are needed.

Adequate functional tests for nestmates will arise naturally from language compliance tests, after the `javac` compiler is modified to take advantage of nestmate access.


Risks and Assumptions
---------------------

The new rules will have to be associated with a new class file version number, as we need to ensure that a Java source compiler only generates bytecodes relying
on the new attributes and rules, when targeting a JVM that understands them. A corollary to that is that a JVM will only recognize and act upon the new attributes, if they appear in a classfile with a suitable version number. A new classfile version imposes a burden on the tools in the broader Java ecosystem, but we do not expect Nestmates to be the sole technology, in the targeted JDK release, that will be relying on a new class file version number.

Loosening access presents little conformance risk. All Java language accesses that compile and run today, will compile and run with the nestmate changes, with no changes to the source code. Code that disables access checking (via `setAccessible`) for reflective access to nestmates today, will continue to run correctly with the nestmate changes - but can be updated to not disable access checking.

Compliance tests that check for prohibited behaviour can fail is some cases.  For example:

- Direct reflective access to private nest methods currently fails (unless access checking is disabled) but will "unexpectedly" succeed with these changes applied.
- A test that `invokeinterface` can't be used for a private interface method will now fail because with these changes it can be used.

There is little or no risk to user compatibility, since the proposal
loosens access.  However, if users have "discovered" and exploited
access bridge methods, they will be unable to do so after the bridges are
dropped.  Such a risk is very small, since bridge methods do not have
stable names in the first place.

There is little or no risk to system integrity, since the proposed
rules confer new access only within a single runtime package.  By
removing the need for bridge methods, potential access between
distinct top-level classes will be systematically decreased.

Nest membership validation requires the presence of the nest-host class, even if that class is itself unused (except as a container for the nest members). This can have impact in three areas:

 1. The order of class loading can change as the nest host may be needed for an access check earlier than when any direct use of the nest host occurs. This is not expected to be a problem as the class is only loaded, not initialized, and dependencies on class loading order (as distinct from class initialization order) are very rare.

 2. This can impact tests/applications that strip out unused classes from their distributed forms, and the nest host is unused. By leaving nest membership validation until the time that a nestmate access check is needed we aim to minimize the impact of this issue, but in some cases the end user will have to change the way they distribute their code. We think this is a very small risk as it isn't common to use a top-level class purely as a stateless container, containing only static nested types, where the nested types will rely on private access to each other.

 3. Resolution of the nest host also introduces class loading (and the potential for associated exceptions) into the JVM's access checking logic. This is primarily a concern for JVM implementors. Care must be taken to ensure all paths that can lead to a VM access check either preclude the possibility of loading a nest host, or else can cope with it. Similarly for the potential exceptions that can occur. From the user perspective there is very little risk due to this as Java code rarely makes assumptions about when and where class loading may occur, and exceptions will only occur if there are badly formed class files.