JDK-8171335 : MethodHandle.Lookup functionality to define a nestmate class
  • Type: Enhancement
  • Component: core-libs
  • Sub-Component: java.lang.invoke
  • Priority: P4
  • Status: Resolved
  • Resolution: Duplicate
  • Submitted: 2016-12-16
  • Updated: 2020-04-21
  • Resolved: 2020-04-21
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 15
15Resolved
Related Reports
Blocks :  
JDK-8227415 - [nestmates] IllegalAccessError using a protected method reference declared in super class in different package
Duplicate :  
JDK-8238358 - Implementation of JEP 371: Hidden Classes
Relates :  
JDK-8230501 - Class data support for hidden classes
Relates :  
JDK-8243287 - Removal of Unsafe::defineAnonymousClass
Relates :  
JDK-8098817 - Adjust access checks for VM anonymous classes
Relates :  
JDK-8213402 - [Testbug] java/lang/System/LoggerFinder/internal/PlatformLoggerBridgeTest creates an invalid nest relationship
Relates :  
JDK-8235521 - Replacement API for Unsafe::ensureClassInitialized
Relates :  
JDK-8213465 - Type variable information is not always maintained for anonymous classes
Relates :  
JDK-8230502 - Add support in JVM TI, JDI, and Instrumentation for hidden classes
Relates :  
JDK-8212620 - Provide a mechanism to allow a class/method to request filtering from the stack trace
Relates :  
JDK-8172672 - Unsafe.defineAnonymousClass should not require non-null host class argument
Relates :  
JDK-6653858 - dynamic languages need to be able to load anonymous classes
Relates :  
JDK-8209676 - Dynamic nestmate injection in the VM
Relates :  
JDK-8219612 - [TESTBUG] compiler.codecache.stress.Helper.TestCaseImpl can't be defined in different runtime package as its nest host
Description
The method Unsafe.defineAnonymousClass:

    public Class<?> defineAnonymousClass(Class<?> hostClass, byte[] data, Object[] cpPatches) {

is used in a few areas to define anonymous classes such as LambdaMetaFactory (although only one use leverages the constant pool patch ing).

The defining of a class (anonymous or otherwise) within the context of a hosted class can be referred to as defining a "nest mate" class.

A subset of such functionality could be supported as a method on MethodHandle.Lookup:

  public Class<?> defineNestmate(boolean anonymous, byte[] bytes)

Where the lookup class is the context for the nest, and where the method only functions if the lookup supports private access.

Code in LambdaMethodFactory and StringConcatFactory could be updated to use the public method.
Comments
This RFE can be closed as a dup of JDK-8238358. LambdaMetaFactory and StringConcatFactory have migrated away from VM anonymous and uses the new `Lookup::defineHiddenClass` API. `Unsafe::defineAnonymousClass` is deprecated and will be deprecated for removal when JDK-8230501 is resolved. JDK-8243287 tracks the relevant tasks related to VM anonymous class.
21-04-2020
The ClassLoader doesn't work as a mailbox for the classData because one CL might be concurrently in use during the loading of many HCs. The ThreadLocal technique is fragile in the ways Remi describes. If the <clinit> code gets executed at a surprising time, the mailbox is not present and the classData is dropped. The ClassValue technique is OK but has a lot of storage overhead (and data motion) to deliver the classData "mail". The overheads remain indefinitely (unless we define a way to clear the CV after a one-time use). Having the JVM "carry the mail" from L.define*Class up to HC.<clinit> requires an extra word associated with the class mirror for the HC. Although this requires some JVM engineering, I think it is the most effective way to get the mail delivered. As an option, the JVM can (if we choose) ensure that the pointer is discarded or nulled out after the class's <clinit> is called, and/or ensure that the query function can only be executed once. Such rules can be created on top of CVs but with significantly more simulation overheads. I view the CV as the prototype and the JVM hardwire as the proper implementation. Some extra usage notes regarding classData: Frameworks which dynamically create classes sometimes need nestmate access to requesting classes, which is why we allow nest injection. Independently, frameworks which are trusted to use special capabilities, such as unsafe or privileged access to JVM resources, sometimes need to inject references to those capabilities, even when the requesting class has no such access. This is why we also allow classData as a way to inject arbitrary information into a hidden class, independently of its access status relative to its lookup class. In effect, a hidden class may be called to merge into the lookup class nest, and also carry secrets hidden from that nest. Also, frameworks should not to use private static finals (in the HCs they spin) to hold secrets, if they also use nest injection. The reason for this is that a nestmate of the HC may obtain access to such a private static final and observe the framework's secret. (This can't happen symbolically via a normal "getstatic" instruction but can happen due to reflection, if the nestmate acquires a reference to the class mirror of the HC.) Thus, HCs which contain framework secrets *and* are nest-injected should use condy constants (or some other non-field-based formulation) to store their secrets. The above consideration puts a requirement on the classData access API for Lookup. It means that the Lookup provided to fetch a classData must be exactly a lookup obtained by MethodHandles.lookup (which is caller-sensitive) and *not* the result of any Lookup-valued transform (even if the result has PRIVATE and MODULE privileges, with all the rest). Enforcing this rule may require the introduction of a new access mode bit, which could be called ORIGINAL, meaning that the Lookup is an original (from MHs.lookup() or a BSM call) and not a teleported copy, with downgraded access.
14-11-2019
The question is how to hand of the value from where you are reifying the Class from the bytecode representation by calling defineClass to the bootstrap method called by constantdynamic. I've used two ways: - You can store the value in the ClassLoader and but you have to control of its code. - call a static method that will store the value in a ThreadLocal, do a ldc on condy that will retreive the value and then clear the ThreadLocal. This works but tend to initialize the class too eagerly and doesn't work if the class has a static block. To avoid the class initialization, i'm using a ClassValue, hand of the value to the ClassValue and get it from the ClassValue when the bootstrap of condy is called. The main issue here is that the class created by defineClass may not have access to the class that store the ClassValue. But if the ClassValue is stored inside java.base, it should be ok.
12-10-2019
Without (4), can you achieve that by using condy?
12-10-2019
Hi Mandy, given (4), i can initialize 1 constant dynamic with the live constant so yes, this is equivalent to do the constant patching of defineAnnonymous. About the VM internal annotations, the only one i really miss is @Hidden, i can more or less simulate the others using method handles, because without it the adapter code/proxy code will not be hidden from the end user :(
11-10-2019
[~forax] w.r.t. your use of constant pool patching, can your proxy 2 library use constant dynamic? Yes, it's true that it requires to spin more BSMs. I'm trying to understand if constant dynamic satisfy the common use cases.
11-10-2019
We categorize the replacement for `Unsafe::defineAnonymousClass` with the following functionalities: 1. hidden class. Hidden classes are non-discoverable by bytecode instructions in other classes and reflection 2. weak class. Weak classes are weakly referenced by its defining class loader. Hence it can be unloaded while its defining loader is still alive. 3. dynamic extension of an existing nest with new nest members 4. ability to attach live constants (JDK-8230501) to a class at define time. Proposed API and specdiff: http://cr.openjdk.java.net/~mchung/valhalla/webrevs/8171335/api/ http://cr.openjdk.java.net/~mchung/valhalla/webrevs/8171335/specdiff/ VM internal annotations `@ForceInline`, `@DontInline` and `@Stable` provide the ability to control JIT optimization. Supporting these VM annotations is out of the scope of this RFE while some may be covered by future enhancements (http://openjdk.java.net/jeps/8209964)
10-10-2019
[~acorn] Correction: I was susceptible of the use case of defining a NON-FINDABLE class but not TEMPORARY. But I do agree the need to request a class to be TEMPORARY so that it can be GC'ed independent to the liveness of its defining class loader. This just gives a choice how the API can be defined.
22-08-2018
Hi Karen, If TEMPORARY is not supported, it means that we still have to generate one ClassLoader per class, which is a memory hog for dynamically typed langages that are not class based, smalltalkish languages like Self, IO, JavaScript, Ruby, etc. For the use of internal annotations, i think you misunderstood why i mention those annotations, i do not want to access them, i want the dynamic nestmate to have the same semantics if possible. By example, if a dynamic nestmate is created as NON-FINDABLE, it should be not visible in the stacktrace (except in a crash dump) so i don't need to access to the annotation @Hidden. Also, obviously, a NON-FINDABLE nestmate is not Serializable so all its final fields are truly final. Along the same idea, if there is a flag TRAMPOLINE (or any other name) that guarantee that all the methods are inlined (it's not real codes it's just a trampoline codes), i don't need to access to @ForceInline.
21-08-2018
1. Mandy does not want to expose the "TEMPORARY" concept via the APIs, since it is working around an implementation challenge. So just the one concept of NON-FINDABLE - name TBD which implies the class can not be found via: Class.forName(), ClassLoader.findClass(), bytecode resolution, ClassFileLoadHook, redefineclass/retransformclass 2. Request for use of jdk internal annotations: Annotations are for JDK Internal APIs and not exported for external use based on strong encapsulation. Access to jdk.internal.vm.annotation.ForceInline is permitted if jdk.internal.vm.annotation package is exported to caller���s module. Use ���add-exports.
21-08-2018
Karen, for the point 3: Who can get the userData? Anyone who gets their hands on the Lookup? Or just the class containing the userData? The lookup is a mechanism to delegate the rights of a class to another one, so anyone who get the Lookup (in PRIVATE mode) has acess to the user Data. for point 4: Ensure that we don't unload the userData while the defined class is still live, and that the UserData life does not keep the class alive, a user Data is like a static field but its value is know when the class is defined and not when the class is initialized (when the static block is run). A related question: is is it possible to access to the userData without initializing the class, and the answer should be yes.
16-04-2018
Couple of notes/questions from EG conversation and follow-up thoughts: 1. NESTMATE is orthogonal to UNNAMED/TEMPORARY. This allows creating nestmates that are named. It also allows creating classes that are not nestmates that are unnamed and even temporary as a general capability. You need to specify each of the attributes you want. 2. UNNAMED class can not be found in dictionary or looked up via reflection. UNNAMED class has the same lifecycle as class loader. 3. TEMPORARY class MUST also be UNNAMED and allows a different lifecycle than the class loader. Note, for Hotspot there is additional overhead for being a TEMPORARY class, so recommend that you don't use it in a situation in which you for example immediately store the class in a strong reference held by the creating class. 2. Need to clarify the Lookup mode requirements 3. Who can get the userData? Anyone who gets their hands on the Lookup? Or just the class containing the userData? 4. Ensure that we don't unload the userData while the defined class is still live, and that the UserData life does not keep the class alive 5. Not part of the discussion: Can we require that a nest host can not be a TEMPORARY class? No problem with UNNAMED - it is a concern about changing access controls by reducing them dynamically. Or with dynamic nestmates since we always have a pre-resolved nest-host, perhaps we can keep the nest-host alive as long as any nestmates are live? 6. Request for getNestMembers to return static and dynamic nest members.
16-04-2018
annotations: - java/lang/invoke/LambdaForm$Hidden is the most important, i use it to generate proxies that are not visible in the stack trace (like the lambda proxy generated by the LambdaMetaFactory). - jdk/internal/vm/annotation/ForceInline because i want to be sure that the caller and the callee are merged together, when you know that inlining the callee will help to simplify the caller code. I have also used DontInline and Stable, but less frequently. but annotations are only one part of the privileges, the other part is the fact that final fields in an anonymous class are trusted [1], thus if the instance of an anonymous class is a constant, the values of the final fields are constant too. [1] http://hg.openjdk.java.net/jdk/jdk/file/49489/src/hotspot/share/ci/ciField.cpp#l204
16-04-2018
Notes from Remi at March 28, 2018 Valhalla EG: Current Unsafe_DefineAnonymousClass today allows vm anonymous classes to be "privileged" in the sense of supporting annotations that require privileges. One that was mentioned was @forceinline for performance. Request for Remi and other user: Which annotations are you currently use and would like in the new API? And why do you need them?
16-04-2018
For nest injection, you need a PRIVATE mode Lookup for a member of the nest. That counts as your authorization to create a dynamic nestmate. Note that reflection will only return static nestmates, but will return true for areNestMates(). Today creating a class in the same package requires PACKAGE mode. In both cases, the new class is created with the same package, module, class loader and protection domain. Question: Would it make more sense to require a PRIVATE mode Lookup for the NestHost explicitly, rather than for any member of the nest? The point is to have the NestHost provide authorization dynamically analogous to the classfile static NestHost authorization.
09-03-2018
More details on the classData proposal: The goal is to allow initialization of a conceptual single private static final field for the new class prior to execution of <clinit>, which is unnamed. The type of the conceptual field is java.lang.Object, so it can contain any Object instance. There is NO constant pool which represents this field, so not way to directly access it via bytecodes or reflection, MethodHandles, VarHandles, etc. This field will not be represented in the ConstantValue attributes. A <clinit> or any other method in the new class or in a nestmate can read this value only via Lookup.findclassData(), so via a condy BSM. There is no way to write this field, even via SetAccessible(). Eg. from an email from John here is an example BSM: public static <T> T classData(Lookup lookup, String ignore, Class<T> type) { Object cd = lookup.findClassData(lookup.lookupClass()); return type.cast(cd); } Note: the Lookup here is created for the condy and is a Lookup for the class which is calling the BSM, i.e. the newly defined class, not the creator of the class.
05-02-2018
Request for adding another flag: TEMPORARY (language choice of name of course) which implies a class lifecycle which is not tied to the definer's class. A TEMPORARY class must also be UNNAMED, but not the inverse. This gives the JVM a head's up on optimizations to allow independent class unloading, and allows the JVM to not waste potential extra overhead (e.g. memory) for classes that actually are tied to the definer's class - e.g. if the newly created class has a BootstrapMethod that returns a MethodHandle for a method in the definer's class.
05-02-2018
Another sketch, as an omnibus API, after discussion (Feb 2017, Jan 2018): class Lookup { /** Load a class, using the permissions of this Lookup. * If the UNNAMED flag is present, the class cannot be found by Class.forName. * If the NESTMATE flag is present, the class is injected into the nest of the lookupClass. * Relevant permissions are checked on lookupModes, especially for nest injection. * @param classFile a sequence of bytes giving a class-file image * @param flags a bit-encoded combination of UNNAMED and/or NESTMATE * @param classData if present and non-null, provides a value which findClassData can retrieve */ Class<?> defineClass(byte[] classFile, [int flags, [Object classData]]); public final static int UNNAMED = 0x01, NESTMATE = 0x02; /** Retrieve the non-null classData argument passed when the given declaringClass was loaded. * If none was supplied, return null. If the current Lookup does not have private access in declaringClass, * throw an access error, since this would be the equivalent of reflectively accessing a private static field. */ Object findClassData(Class<?> declaringClass); } More discussion: A known-good substitute for CP patches is to allow the caller of defineClass to pass exactly one extra parameter (an Object) which the defined class gets private access to. That's just an API something like this: class Lookup { Class<?> defineClass(String name, ByteSequence classFile); // simplest point Class<?> defineClass(String name, ByteSequence classFile, Object classData); Object findClassData(Class<?> declaringClass); // returns null if none was supplied // and for nestmates: Class<?> defineClass(Class<?> nestHost, String name, ByteSequence classFile); // nestmate Class<?> defineClass(Class<?> nestHost, String name, ByteSequence classFile, Object classData); } (I'm slyly mentioning ByteSequence instead of byte[], since I think we someday want such a type, for immutable opaque byte sources. And then we can hook them to CONSTANT_Bytes CP entries. Just read it as byte[] if you don't want to think about that.) The classValue feature isn't particularly hard compared to the other bits we are talking about. The special part is plumbing the classData up through the JVM's class loading code. Doing the nestmate hookup requires almost the same plumbing. This replaces the old hack of patching the constant pool, and makes it obsolete. If you need N "live constants" for a defineClass class, you first pack them into a single Object (List.of, anyone?) and pass it as the classData of that class. When you need to use it from inside the class, first you get ahold of the classData, and then select which of the N constants you want. You might choose to make N static finals and set them all up in the <clinit>, or you might just hold onto the classData as a whole: User's choice. The key is that a class can always see its own classData, which acts like an unnamed private static variable in each class. (In the common case of null, no field allocation is needed, of course.) The main (or only) use of the classData would be to get it in the class's <clinit> method and use that to initialize the class's internal static finals. The findClassData could also be getAndClearClassData, on the theory that these things are only useful during bootstrapping, and should be cleared away afterward. Obviously the only safe place to use it is in a <clinit> method for the class holding the data. A workaround for that is for a user to create a new List<?>[1] box to hold the classData, and patch it to null in the <clinit> after the value is extracted, so clearing is not necessary to the user model. The classData could be hosted on ClassValue internally. I don't see any API design benefit from exposing a ClassValue per se, except maybe to allow control of clearing. So maybe: class Lookup { static ClassValue<Function<Lookup,Object>> classDataKey(); } But it doesn't gracefully support the security check (hence the indirection through the function type).
11-01-2018
A few more thoughts: This functionality could be created *almost* above the level of the JVM by using a null host class and loading the class into the target package. Currently, though, the JVM doesn't tolerate null host classes for anonymous classes. I think this is straightforward but a little messy to do, and is a reasonable thing to experiment with. If the Lookup object does not have the PRIVATE bit set, it mustn't be allowed to inject a nest mate. But if it has PACKAGE set, it is reasonable to inject a package member alongside the lookup class. Package-level hosting could be emulated at the library level, with no JVM changes, by doing a one-time injection of a hidden empty class HEC(P) into each package, and then using HEC(P) as the implementation's host class for all package level anonymous classes. This would be a fine way to prototype a new API without entangling the JVM. The final version should make a small cut into the JVM to disentangle the host class from anonymity. Likewise, a replacement for constant pool patching could be defined as the API mentioned in the previous comment so a lookup-loaded class can query, within its static initializer, a special value (or array of values) proposed by the caller of the lookup-loading API. There is no clean way to do this today, but it is easy to mock up as a prototype, using thread locals and try/finally. Eventually the JVM could take part in passing the extra "user data" parameter(s) through from caller to <clinit>. Main point: We should prototype a library-only version of L.dAC and friends, shake out the use cases and design flaws, and eventually settle it into the JVM. The combined signature would be: Lookup.defineClass(byte[], optional boolean isAnonymous, optional Object initializationContext); (Optionals suggest telescope points for extra overloads.) The new class has access to the nest of the lookupClass if private access. Otherwise, it has access to the package of the lookupClass if package access. Weaker access mode cases cause errors and are reserved for future use. The initializationContext is available during execution of the new class's <clinit>. If the iC is non-null, then initialization takes place before defineClass returns. Otherwise we can talk about deferring initialization. (Currently it is eager for U.dAC.) This context value can carry "live values" which are currently carried by the constant pool patching mechanism, and can install them as effective constants in private static final variables. These can be method handles, lookup objects, arbitrary user objects, or collections of all of the above. (Clearly there are up to four entry points here: dC, dAC, dC_IC, dAC_IC. I have no opinion on how they should telescope.) TBD: What happens if the proposed class is named with the wrong package? Can we just outlaw this? We probably do need to outlaw it if the class is named. If it is anonymous, we could require that its name is in the *empty* package, for regularity, and ask the JVM to supply a package prefix. But it seems simpler to move towards requiring well-behaved package prefixes, for all cases. For capturing the injected static context, while <clinit> is running, we need something like: /** Return the context specified when this class was defined. * Must be called during execution of that class's static initializer, * in the same thread that is running the initializer. * Throws ISA if not called during the static init (same thread), * or if the context was specified as null or was not specified, * or if this class was not loaded via * an API that supports a context parameter. */ Object Class.getInitializationContext() throws IllegalStateException;
18-02-2017
I agree with Paul that constant patching is the wrong thing to specify and with Remi that there is a place for such a feature. Three possible ways forward, all of which replace a set of constant pool patches with another dictionary of ad hoc constants, with a different kind of key: 1. [static field patching] Specify a set of <String, field-value> pairs where each key locates a field defined by the class to be loaded. The JVM contrives to initialize each specified field to the specified value. A structural rule requires that the field is static and final (typically private but no need to specify). User code just grabs the final variable, and it will be optimized well by existing JIT tactics. Odd interaction with JLS: If the bytecodes derive from Java source, there is a competing initialization for the field in <clinit>. 2. [class value patching] Specify a set of <cv-key, cv-value> pairs where each key is a previously constructed ClassValue. The JVM contrives to short-circuit the ClassValue bootstrap sequence by pre-binding the new Class to the given value, for the given key. User code calls ClassValue.get on this-class (which BTW gets internally patched, as a constant pool reference). For full optimization, probably requires the JIT to make special rules for ClassValue. 3. [new context API] Specify a single value <ac-context> of type Object when loading the new class, and a new accessor Class::getAnonymousClassContext. The accessor would return null for all non-anonymous classes, and the given (untyped) object for an anonymous class. Based on the user requirements, it could be an array, a Map, a Supplier, a Function, or a random single value. The JIT would intrinsify this function, like it does for many similar ones in java.lang.Class. I think I prefer the third. Makes the trick explicit and avoids accidents.
09-01-2017
A public mechanism to also support constant pool patching of live objects is a more involved feature and may require a more formal mechanism for representing "dynamic" constants. I am uncertain about the safety aspects of the current implementation. I suppose at some level injecting live objects into the constant pool is no different to using explicit static final fields (whose contents should be held constant). At the moment there is a "gentleman's agreement" that strings in the constant pool are replaced with the corresponding object instances via a cast after the ldc, and the verifier is non the wiser to the shell game going on. While that assumption is ok for our implementation i am reluctant to expose that as the public mechanism.
16-12-2016
Why not allowing constant patching too ? Constant pool patching is not unsafe and allow to do type specialization in dynamic languages.
16-12-2016