JDK-8160849 uncovered some ugliness in the ClassFileTransformer.transform() support when transform() makes a reference (directly or indirectly) to the class being transformed. This results in a "circular" reference, although not circular in the ClassCircularityError sense, which happens when there is circularity in the class hierarchy. The JVM currently throws CCE in some cases (which should not be thrown unless the class hierarchy is circular), or throws a LinkageError, which is probably the correct exception, but has a misleading message. More details on the JVMs error handling is given below.

The JLI spec says nothing about what happens if the transform() method triggers additional class loading, possibly referencing the the class being transformed.

https://docs.oracle.com/javase/8/docs/api/java/lang/instrument/ClassFileTransformer.html

The spec should probably should clarify this situation. These circular references put the JVM in a difficult position, and probably the best it can do is throw some meaningful exception.

Note that JLI does prevent recursive invocation of transform(). See the call to tryToAcquireReentrancyToken in jdk/src/java.instrument/share/native/libinstrument/JPLISAgent.c, which disables recursive invocation of transform(). I think it's purpose might be to avoid the CCE situation, but it's not 100% safe. 

Attached to this CR is a test case which demonstrates the JVM behavior with 3 separate "circular" scenarios, each which the JVM handles differently. The test is not written to detect incorrect behavior and. It simply triggers the circular references, and prints out the exceptions thrown by the JVM. Below is a description of the 3 failures it triggers:

Scenario #1: The loading of a classpath class is triggered. When the JVM attempts to load  its superclass, transform() is called for the the superclass. From transform() a reference to the subclass is made. The JVM will try to load the subclass because it is not yet fully loaded. This results in a ClassCircularityError when the JVM sees it is already loading this class. This exception is invalid since ClassCircularityError should only be used when there is a circularity in the class hierarchy, which is not the case here. A LinkageError is likely the correct exception.

Note that the CCE for this test case is thrown by SystemDictionary::resolve_super_or_fail():

      PlaceholderEntry* probe = placeholders()->get_entry(p_index, p_hash, child_name, loader_data);
      if (probe && probe->check_seen_thread(THREAD, PlaceholderTable::LOAD_SUPER)) {
          throw_circularity_error = true;
      }

Scenario #2: The loading of a classpath class is triggered. When the JVM attempts to load the class, transform() is called for it. From transform() a reference to the class is made. The JVM will try to load the class because it is not yet fully loaded. This results in a LinkageErorr. This is probably the correct exception to throw. However, the message with the LinkageErorr states "attempted duplicate class definition", which is misleading. Also, the reference to the class from transform() actually succeeds, resulting in the class being fully loaded at that time. The LinkageErorr is not thrown until transform() returns and the initial loading of the class resumes. This means the initial attempt to load the class fails rather than the attempt made by the transform() method. This seems incorrect. since it is transform() that is behaving incorrectly, not the initial attempt to load the class. It also leaves you with the situation where if the initial attempt to load the class is reattempted, it will succeed because the class is already fully loaded by then.

Note the LinkageError is thrown by SystemDictionary::check_constraints() when it determines that the given class has already been loaded. This is another indication that this scenario is not being handled properly, since the "duplicate" class here has nothing to do with loader constrains. It has to do with class loading code not recognizing that transform() triggered loading the class it was currently loading (or probably more correctly that transform() was able to trigger loading the class without getting an error).

Scenario #3: Same as (2) except the class is on the bootclasspath. The JVM handles this differently, and it results in ClassCircularityError being thrown when transform() references the class. An exception is appropriate here, but just as with (1), it probably should be LinkageErorr, not ClassCircularityError. The CCE is detected by the following code in SystemDictionary::resolve_instance_class_or_null():

          if (oldprobe->check_seen_thread(THREAD, PlaceholderTable::LOAD_INSTANCE)) {
            throw_circularity_error = true;
          } else {