JDK-8336296 : ClassFileTransformer reentrancy and exception handling not specified
  • Type: Bug
  • Component: core-svc
  • Sub-Component: java.lang.instrument
  • Affected Version: 5.0,24
  • Priority: P3
  • Status: New
  • Resolution: Unresolved
  • Submitted: 2024-07-12
  • Updated: 2024-07-12
Related Reports
Relates :  
JDK-8335619 - Add an @apiNote to j.l.i.ClassFileTransformer to warn about recursive class loading and ClassCircularityErrors
Relates :  
JDK-8164165 - JVM throws incorrect exception when ClassFileTransformer.transform() triggers class loading of class already being loaded
Description
ClassFileTransformer does not specify what happens when code executing in the transform method triggers class loading. Long standing behavior (since JDK 5) is that the ClassFileTransformer is not called if a transform is in progress. 

Additionally, ClassFileTransformer does not specify what happens if a transform method completes abruptly with an error or exception (a LinkageError due to recursive class loading for example). Long standing behavior is to silently swallow the exception, and "keep on trucking" by invoking the next transformer in the pipeline. This needs to be specified and potentially re-examined too as it would be a lot better to abort the pipeline and propagate an error/exception when initiated by a retransformClasses or redefineClasses (not much can be done for the initial class load). The compatibility impact of changing this behavior would of course need to examined to see how feasible it is.
Comments
While we're on this, I think it might be worth while to evaluate the possibility of treating resolution errors caused by recursive class loading during transformation differently from ordinary resolution errors, such that they will not lead to a permanent resolution errors for the corresponding constant pool entries even after transformation. This would currently violate section ยง 5.4.3 Resolution (https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-5.html#jvms-5.4.3) of the JVM specification which mandates that "if an attempt by the Java Virtual Machine to resolve a symbolic reference fails because an error is thrown that is an instance of LinkageError (or a subclass), then subsequent attempts to resolve the reference always fail with the same error that was thrown as a result of the initial resolution attempt". But if such resolution errors only occur because of improper usage of class file transformation I think it might be reasonable to retry it after transformation. Alternatively, we could throw an error which is not a subclass of LinkageError in the case of recursive class loading to avoid messing around with the JVMS (also see JDK-8164165 for a discussion about which is the correct error to throw.).
12-07-2024