JDK-8038468 : java/lang/instrument/ParallelTransformerLoader.sh fails with ClassCircularityError
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: hs25,8u40,9
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: windows_8
  • CPU: x86
  • Submitted: 2014-03-27
  • Updated: 2015-06-04
  • Resolved: 2014-11-27
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 9
9 b44Fixed
Related Reports
Relates :  
Description
Starting test with 1000 iterationsThread 'DestroyJavaVM' has called transform()
Thread 'DestroyJavaVM' has called transform()
result=1
----------System.err:(14/920)----------
Exception in thread "main" java.lang.ClassCircularityError: sun/misc/URLClassPath$JarLoader$2
	at sun.misc.URLClassPath$JarLoader.checkResource(URLClassPath.java:771)
	at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:843)
	at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:364)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:426)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:359)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:340)
	at ParallelTransformerLoaderApp.loadClasses(ParallelTransformerLoaderApp.java:83)
	at ParallelTransformerLoaderApp.main(ParallelTransformerLoaderApp.java:45)
Comments
new fix: check if loader is null before loading TestClass3. Remove block for sleep which also is caused by loading loader itself. *** old/test/java/lang/instrument/ParallelTransformerLoaderAgent.java Tue Nov 25 08:07:46 2014 --- new/test/java/lang/instrument/ParallelTransformerLoaderAgent.java Tue Nov 25 08:07:46 2014 *** 77,104 **** --- 77,89 ---- ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException { String tName = Thread.currentThread().getName(); // In 160_03 and older, transform() is called // with the "system_loader_lock" held and that // prevents the bootstrap class loaded from // running in parallel. If we add a slight sleep // delay here when the transform() call is not // main or TestThread, then the deadlock in // 160_03 and older is much more reproducible. if (!tName.equals("main") && !tName.equals("TestThread")) { System.out.println("Thread '" + tName + "' has called transform()"); try { Thread.sleep(500); } catch (InterruptedException ie) { } } ! // load additional classes when called from other threads ! if (!tName.equals("main")) ! // load additional classes when called from thread TestThread ! if (tName.equals("TestThread") && loader != null) { loadClasses(3); } return null; }
25-11-2014

When checked with 7u60, I thought it passed after 3 hours running with no failure, then move to 7u65 b01, it failed after 3 hours. This made me think of the test, in the test case we loop 2000 times (in nightly). In above test, a infinite loop launch the processes (jobs) again and again until a test job failed, it stops but the other jobs still keep going on. So I went back to test 7u60, after more hours, it failed! So this made me think this test case is not a good test case for testing purpose, we cannot make sure there is no problem for the 2000 loops even it passed. Even after hours running without failure, we still CAN NOT make sure it passed! It may fail if the test continue. My guessing is that it will fail given enough test time. The original design goal for this test case is to check the fix of JDK-5088398, the dead lock issue, but in the failed case, we saw that the design has a flaw --- it tries to load another class with the same loader in agent in transformer. The CCE seen in other thread, is from this part, ConstantPool:klass_at_impl(...) Thread A, check if (this_cp->tag_at(which).is_unresolved_klass_in_error()) { // The original attempt to resolve this constant pool entry failed so find the // class of the original error and throw another error of the same class // (JVMS 5.4.3). // If there is a detail message, pass that detail message to the error. // The JVMS does not strictly require us to duplicate the same detail message, // or any internal exception fields such as cause or stacktrace. But since the // detail message is often a class name or other literal string, we will repeat it // if we can find it in the symbol table. throw_resolution_error(this_cp, which, CHECK_0); ShouldNotReachHere(); } It has no error now, but we need to call resolve_or_fail and with CCE return. It call ConstantPool::save_and_throw_exception, since CCE is sub of LinkageError, we call SystemDictionary::add_resolution_error(this_cp, which, error, message); and return 0. (Not clear exception). Now, thread B comes, and resolve same class, since it is added as unresovled, it will throw the original Exception. The code here is 9, in 8, there is a var 'in_error', we throw with in_error set.
21-11-2014

Have done changes to test case, diff: @@ -96,7 +99,10 @@ } // load additional classes when called from other threads - if (!tName.equals("main")) + if (tName.equals("TestThread")) { loadClasses(3); } @@ -106,10 +112,12 @@ public static void loadClasses( int index) { - ClassLoader loader = ParallelTransformerLoaderAgent.getClassLoader(); + ClassLoader loader = getClassLoader(); try { - Class.forName("TestClass" + index, true, loader); + if (ClassLoader.getSystemClassLoader() != null) { + Class.forName("TestClass" + index, true, loader); + } } catch (Exception e) { The change is that, only in TestThread, we call loadClass ( TestClass3 ), and only if SystemLoader is not null, we do the actual loading. This still failed. Thinking check if system class loader (which loads 'sun/misc/URLClassPath$JarLoader$2') can not prevent such kind of failure so I have done a search of which is the earliest version manifest this failure, so far: passed: 7u40, 7u60 failed: 7u71, 8-b01, 8u40 all, 9 all The failure probably started between 7u60 (exclusive) and 7u71 (inclusive). Other tracing for the exception, in ConstantPool::klass_at_impl, after call resovle_or_fail (this is jdk 9 version, 8 or 7 a little structure different but similar): if (!PENDING_EXCEPTION-> is_a(SystemDictionary::LinkageError_klass())) { // Just throw the exception and don't prevent these classes from // being loaded due to virtual machine errors like StackOverflow // and OutOfMemoryError, etc, or if the thread was hit by stop() // Needs clarification to section 5.4.3 of the VM spec (see 6308271) } else if (this_cp->tag_at(which).value() != error_tag) { Symbol* message = exception_message(this_cp, which, tag, PENDING_EXCEPTION); SystemDictionary::add_resolution_error(this_cp, which, error, message); // CAS in the tag. If a thread beat us to registering this error that's fine. // If another thread resolved the reference, this is an error. The resolution // must deterministically get an error. So why do we save this? // We save this because jvmti can add classes to the bootclass path after this // error, so it needs to get the same error if the error is first. jbyte old_tag = Atomic::cmpxchg((jbyte)error_tag, (jbyte*)this_cp->tag_addr_at(which), (jbyte)tag.value()); assert(old_tag == error_tag || old_tag == tag.value(), "should not be resolved otherwise"); } else { // some other thread put this in error state throw_resolution_error(this_cp, which, CHECK); } The trace showed all test loop, we have once call SystemDictionary::add_resolution_error(this_cp, which, error, message). For the failure case, the throw DOES NOT come from call throw_resolution_error here ( jdk8 version). Haven't checked 9 and 7, though it is similar. Will go on the matrix testing to find the specific version which showed the earliest failure.
17-11-2014

I think there are three things we need to figure out. 1. I reproduced a problem in TestThread2. Here is the information from that and my analysis -- please try the suggested test change below to see if it helps. 2. Does the circularity error actually occur in the main thread and if so why? - need to catch in a debugger/hs_err file a situation in which this occurs in the main thread please. We need the full stack trace for this - native and java please 3. figure out why we we see this problem more frequently - I am not convinced this problem didn't already exist - the test logic has some very odd comments and workarounds which seem to imply there were intermittent problems from the beginning - that said - worth figuring out if for instance, the sun.misc.URLClassPath logic was rewritten (and when) to add $JarLoader$2 ---- I ran -XX:AbortVMOnException=java.lang.ClassCircularityError -XX:+ShowMessageBoxOnError to get a log file and a stack trace. I attached a debugger, but it wasn't enough help since I needed to see the java stack frames so I got the stack frames for java from the hs_err_log. The stack trace was on Thread 2, which in the hs_err_log was TestThread (which makes sense for what the test logic says). See later in email for stack traces from Thread 2. Summary of stack trace: TestThread: loadClasses(#) -> forName(TestClass#, URLClassLoader) vm calls out to URLClassLoader.loadClass(String) which is inherited from java.lang.ClassLoader.loadClass(String) ... calls java.net.URLClassLoader.findClass(...) which calls DoPrivileged java.net.URLClassLoader$1.run which calls sun.misc.URLClassPath.getResource(name, false) which calls sun.misc.URLClassPath$JarLoader.getResource which calls sun.misc.URLClassPath$JarLoader.checkResource which tries to call sun.misc.URLClassPath$JarLoader$2 - and then the transformer jumps in with loadClasses(# (which we know is 3) and walks the same logic which tries to load sun.misc.URLClassPath$JarLoader$2 again Note that in the placeholder table information that Yumin printed, the circularity error is on sun.misc.URLClassPath$JarLoader$2 with the null == boot loader, which makes sense -- that is the appropriate defining loader, and therefore the one the CFLH would intercept during the defineClass phase. In the sun.misc.URLClassPath.java file, in the class JarLoader, in the method checkResource ... return new Resource() { ... } -- I do not know why that generates sun.misc.URLClassPath$JarLoader$1, $2 and $3 at build time or when that was added. I would guess that is when the bug started happening. When I have a successful run, sun.misc.URLClassPath$JarLoader$2 loads before any TestClass1 loads. My belief is that the point of the test is to test parallel class loading for URL class loaders. I don't think the point is to test the bootstrap class loader, nor to test bootstrapping - i.e. running the agent before we have loaded sufficient classes to allow loading URLClassLoader classes. What I suggested to Yumin that he try would be to change the test to NOT intercept boot loader loads, so that sun.misc.URLClassPath$JarLoader$# can load which will in turn allow classes loaded by a URLClassLoader subclass to load. So that change to the test would be: in TestTransformer: if (loader != null) { if (tName.equals("TestThread")) { { loadClasses(3); } } return null; } // I also suspect with that change, we can remove the sleep loop Note: there was a printed message which said that the Thread "Signal Dispatcher" has called transform(), which I ignored, however it is good that we don't call loadClass on that thread - which is part of what the sleep loop does - but that would be handled by the boot loader screening above Alternatively we can preload the URLClassPath classes, but I don't think we want to do that, or we can have the agent explicitly screen on a variety of jdk bootstrapping classes. But I think the cleaner solution is to screen on the boot loader. Does that make any sense to others? thanks, Karen p.s. How to run with hotspot flags (jtreg has a -show:rerun option, but with a shell script in the test, this is more complex, so the following should be easier): So what I did was run the test once for it to pass (not your script, but just once with jtreg) so that it generated the $DST/work directory. I then created a rerun.csh script - attached - you can modify for your own $DST directory. I used it to be able to quickly rerun the test without the jtreg framework and compile time etc. but mostly to be able to actually add hotspot command-line flags. p.p.s. details from the error log (let me know if you want me to attach the error log to the bug report) note: error log shows last 10 events including: Event: 0.928 loading class sun/misc/URLClassPath$JarLoader$2 Event: 0.928 loading class TestClass3 Event: 0.929 loading class TestClass3 done Event: 0.929 loading class java/lang/ClassCircularityError Event: 0.929 loading class java/lang/ClassCircularityError done TestThread java frames: j sun.misc.URLClassPath$JarLoader.checkResource(Ljava/lang/String;ZLjava/util/jar/JarEntry;)Lsun/misc/Resource;+42 j sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;+54 j sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;+53 j java.net.URLClassLoader$1.run()Ljava/lang/Class;+26 j java.net.URLClassLoader$1.run()Ljava/lang/Object;+1 v ~StubRoutines::call_stub j java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0 j java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13 j java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+70 j java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3 v ~StubRoutines::call_stub j java.lang.Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;Ljava/lang/Class;)Ljava/lang/Class;+0 j java.lang.Class.forName(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;+49 j ParallelTransformerLoaderAgent$TestTransformer.loadClasses(I)V+25 j ParallelTransformerLoaderAgent$TestTransformer.transform(Ljava/lang/ClassLoader;Ljava/lang/String;Ljava/lang/Class;Ljava/security/ProtectionDomain;[B)[B+81 j sun.instrument.TransformerManager.transform(Ljava/lang/ClassLoader;Ljava/lang/String;Ljava/lang/Class;Ljava/security/ProtectionDomain;[B)[B+50 j sun.instrument.InstrumentationImpl.transform(Ljava/lang/ClassLoader;Ljava/lang/String;Ljava/lang/Class;Ljava/security/ProtectionDomain;[BZ)[B+34 v ~StubRoutines::call_stub j sun.misc.URLClassPath$JarLoader.checkResource(Ljava/lang/String;ZLjava/util/jar/JarEntry;)Lsun/misc/Resource;+42 j sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;+54 j sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;+53 j java.net.URLClassLoader$1.run()Ljava/lang/Class;+26 j java.net.URLClassLoader$1.run()Ljava/lang/Object;+1 v ~StubRoutines::call_stub j java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0 j java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13 j java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+70 j java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3 v ~StubRoutines::call_stub j java.lang.Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;Ljava/lang/Class;)Ljava/lang/Class;+0 j java.lang.Class.forName(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;+49 j ParallelTransformerLoaderApp.loadClasses(I)V+25 j ParallelTransformerLoaderApp$TestThread.run()V+4 v ~StubRoutines::call_stub
12-11-2014

From email discussion: I've had a look at the test code and the bug report and I'm still quite confused by the analysis. First I think we can disregard the DestroyJavaVM part of this. The main thread encounters the CCE and so main terminates and we see the "Exception in thread "main" ..." on stderr. That thread detaches and re-attaches as the DestroyJavaVM thread. When the TestThread terminates the DestroyJavaVMThread initiates shutdown and that causes loading of the Shutdown class and that triggers execution of transform(). But this is all well after the CCE happens in main. The problem occurs when we are loading the system classes. I can imagine that the main thread is trying to execute loadClasses(1) which will use the agent defined classloader to try and load TestClass1. In invoking Class.forName it seems that some of the URLClassLoader internal classes - ie JarLoader$2 - still need to be loaded and so we start to load those. Concurrently the TestThread wants to load TestClass2 and so it too has to ensure that all the URLClassLoader classes are loaded, but at some point the transform() is encountered and so it then starts to load TestClass3. This in turn still needs all of the URLClassLoader classes to complete loading. I'm unclear at what point in the loading process transform() gets invoked, but I can easily imagine a cycle (or even an infinite loop!) developing here. But the CCE occurs in the main thread and the main thread doesn't load anything via transform() so I am at a loss to understand how it can encounter CCE. --- I don't think the test has an issue. As Serguei asks, what has changed such that this failure has started to appear?
26-10-2014

Some analysis to make sure I understand the test scenario. The ParallelTransformerLoaderApp.main() executes a 1000 iteration loop. It does at each iteration: - creates and starts a new TestThread - loads TestClass1 with the current class loader: ParallelTransformerLoaderAgent.getClassLoader() - changes the current class loader with new one: ParallelTransformerLoaderAgent.generateNewClassLoader() The TestThread loads the TestClass2 concurrently with the main thread. At the CFLH events, the ParallelTransformerLoaderAgent does the class retransformation. If the thread loading the class is not "main", it loads the class TestClass3 with the current class loader ParallelTransformerLoaderAgent.getClassLoader(). Sometimes, the TestClass2 and TestClass3 are loaded by the same class loader recursively. I'm not sure yet the test is incorrect. It is not clear to me if this failure is a regression. Did we observe this issue before? If - NOT then when and why had this failure started to appear?
24-10-2014

Dmitry, It seems there is a confusion here. I do not know there is a requirement that the nightly failure CRs should be created as confidential. Some comments in the CR need to be confidential because they have links to non-public nightly results. This does not mean the whole CR must be filed as confidential, right? The RFR for this bug must be discussed in public.
24-10-2014

I already set this bug as public, but need to know why at first place it was filed as confidential? Dmitry, do you still remember why you filed as confidential? Thanks
13-10-2014

More trace for jdk9 the most recent test showed it did throw CircularityError with the same reason. Test case did not catch the exception.
10-10-2014

The test case has a problem. There are two threads running in parallel in loading TestClass1 and TestClass2, respectively in 'main' thread and 'TestThread'. Those two threads will all call 'transform' since the class loader of user defined class loader. In 'transform', if thread name is not 'main', load TestClass3 with same class loader as TestClass1 and TestClass2. This will cause Circularity error, after discussed with Karen, we got conclusion that the test case is the main reason for this error. The detailed trace showed result: TestClass3 is loaded in processing of loading of TestClass2. <add_entry new index=424 seen_thread=25857> 'TestClass2', loader class loader 0x00007f632c316620a 'java/net/URLClassLoader' loadInstanceThreadQ threads:AllocatedObj(0x00007f632c314800), superThreadQ threads: defineThreadQ threads: </add_entry> <remove_entry index=505 thread=25857 act=1> 'TestClass2', loader NULL class_loader loadInstanceThreadQ threads:AllocatedObj(0x00007f632c314800), superThreadQ threads: defineThreadQ threads: remove_seen_thread: 25857 entry of index removed!!!! </remove_entry> This uses NULL loader but no found and removed from placeholer, then begins with User loader, since JarLoader$2 is system class, it delegates to NULL loader. There is a lot intermediate steps in lib/vm involved. Here only lists main concerned ones. <add_entry new index=85 seen_thread=25857> 'sun/misc/URLClassPath$JarLoader$2', loader NULL class_loader loadInstanceThreadQ threads:AllocatedObj(0x00007f632c314800), superThreadQ threads: defineThreadQ threads: </add_entry> Then begins loading TestClass3: <add_entry new index=864 seen_thread=25857> 'TestClass3', loader class loader 0x00007f632c316620a 'java/net/URLClassLoader' loadInstanceThreadQ threads:AllocatedObj(0x00007f632c314800), superThreadQ threads: defineThreadQ threads: </add_entry> <add_entry new index=521 seen_thread=25857> 'TestClass3', loader NULL class_loader loadInstanceThreadQ threads:AllocatedObj(0x00007f632c314800), superThreadQ threads: defineThreadQ threads: </add_entry> <remove_entry index=521 thread=25857 act=1> 'TestClass3', loader NULL class_loader loadInstanceThreadQ threads:AllocatedObj(0x00007f632c314800), superThreadQ threads: defineThreadQ threads: remove_seen_thread: 25857 entry of index removed!!!! </remove_entry> Note, index 521 is for NULL loader and no found so removed from PlaceHolderTable. With define class go through the same route as TestClass2, it tried to load JarLoader$2 again in the same thread, same NULL loader, and found in place holder table, set throw_circularity_error and throw CircularityError. There are many cases CircularityError thrown without failing the test case but it is not caught in TestThread, this is not known. If CircularityError got caught (design in main and TestThread is to catch Exception), there should no test failure. The change should be at test case.
09-10-2014

Correct for last upate: it failed when sun/misc/URLClassPath$JarLoader$2 loading twice. The crashed case, failed at get instanceKlass of java/lang/CircularityError after throw it. It looks most of the cases, the CircularityError happed after TestClass3 was loaded (which is in the test case, loading thread is "TestThread") first. In vm I couldn't find places when locking escaped caused inconsistent status. the test code: for (int i = 0; i < kNumIterations; i++) { // load some classes from multiple threads (this thread and one other) Thread testThread = new TestThread(2); testThread.start(); loadClasses(1); // log that it completed and reset for the next iteration testThread.join(); System.out.print("."); ParallelTransformerLoaderAgent.generateNewClassLoader(); } The loading of TestClass<i> is in parallel: loadClasses(1) loads TestClass1 in 'main' thread, testThread loads 'TestClass2', while with 'transform' in agent, TestClass3 only loaded in TestThread: if (!tName.equals("main")) { loadClasses(3); } It happens when resolving 'sun/misc/URLClassPath$JarLoader$2'.
03-10-2014

After adding tracing code, before CircularityError thrown: ..... add_seen_thread: 13053 'sun/misc/URLClassPath$JarLoader$2', loader NULL class_loader loadInstanceThreadQ threads:AllocatedObj(0x00007f3020325000), superThreadQ threads: defineThreadQ threads: add_seen_thread: 13053 'TestClass3', loader class loader 0x00007f30203269f0a 'java/net/URLClassLoader' loadInstanceThreadQ threads:AllocatedObj(0x00007f3020325000), superThreadQ threads: defineThreadQ threads: add_seen_thread: 13053 'TestClass3', loader NULL class_loader loadInstanceThreadQ threads:AllocatedObj(0x00007f3020325000), superThreadQ threads: defineThreadQ threads: remove_seen_thread: 13053 remove entry index = 521 resolve_instance_class_or_null, Current thread: 13053, THREAD: 13053 PlaceholderTable: place holder 'sun/misc/URLClassPath$JarLoader$2', loader NULL class_loader loadInstanceThreadQ threads:AllocatedObj(0x00007f3020325000), superThreadQ threads: defineThreadQ threads: place holder 'TestClass2', loader class loader 0x00007f30203269f0a 'java/net/URLClassLoader' loadInstanceThreadQ threads:AllocatedObj(0x00007f3020325000), superThreadQ threads: defineThreadQ threads: place holder 'TestClass3', loader class loader 0x00007f30203269f0a 'java/net/URLClassLoader' loadInstanceThreadQ threads:AllocatedObj(0x00007f3020325000), superThreadQ threads: defineThreadQ threads: place holder 'TestClass1', loader class loader 0x00007f30203269f0a 'java/net/URLClassLoader' loadInstanceThreadQ threads:AllocatedObj(0x00007f302000b000), superThreadQ threads: defineThreadQ threads: later, it either crashed or CircularityError thrown and exit. Note at crashed case, the thread 0x00007f3020325000 (obj), lwp 10535 already exit so no valid data in memory can be retrieved, this is the thread created for ParallelTransformerLoaderApp$TestThread. TestClass2 and TestClass3 are loaded in this thread. Three classes all loaded with same class loader as design. TestClass3 was loaded twice, one with designed loader, the other with default loader (NULL), and the removal from the placeHolderTable only once. This is why we can see the check for CircularityError is true.
01-10-2014

When tested with a debug version, got a crash. hs_err_pid2678.log attached. The constant pool is corrupted.
25-09-2014

The problem occurs in resolve_instance_class_or_null: PlaceholderTable: place holder 'sun/misc/URLClassPath$JarLoader$2', loader NULL class_loader loadInstanceThreadQ threads:AllocatedObj(0x00007f1248308800), superThreadQ threads: defineThreadQ threads: place holder 'TestClass2', loader class loader 0x00007f124830a4b0a 'java/net/URLClassLoader' loadInstanceThreadQ threads:AllocatedObj(0x00007f1248308800), superThreadQ threads: defineThreadQ threads: place holder 'TestClass3', loader class loader 0x00007f124830a4b0a 'java/net/URLClassLoader' loadInstanceThreadQ threads:AllocatedObj(0x00007f1248308800), superThreadQ threads: defineThreadQ threads: place holder 'TestClass1', loader class loader 0x00007f124830a4b0a 'java/net/URLClassLoader' loadInstanceThreadQ threads:AllocatedObj(0x00007f124800b000), superThreadQ threads: defineThreadQ threads: It happened loading sun/misc/URLClassPath$JarLoader$2, which is the second default method to get JarLoader: newLoader = AccessController.doPrivileged( new PrivilegedExceptionAction<JarLoader>() { public JarLoader run() throws IOException { return new JarLoader(url, handler, lmap); } });
25-09-2014

I could run and got same result. Last part of the log: [Loaded TestClass3 from file:/tmp/yqi/8038468/13.w/scratch/Test.jar] [Loaded TestClass2 from file:/tmp/yqi/8038468/13.w/scratch/Test.jar] [Loaded java.lang.Throwable$WrappedPrintStream from /net/scanas412.us.oracle.com/export/java_re2/jdk/8u40/ea/b06/binaries/linux-x64/jre/lib/rt.jar] [Loaded java.util.IdentityHashMap from /net/scanas412.us.oracle.com/export/java_re2/jdk/8u40/ea/b06/binaries/linux-x64/jre/lib/rt.jar] [Loaded java.util.IdentityHashMap$KeySet from /net/scanas412.us.oracle.com/export/java_re2/jdk/8u40/ea/b06/binaries/linux-x64/jre/lib/rt.jar] Thread 'DestroyJavaVM' has called transform() [Loaded java.lang.Shutdown from /net/scanas412.us.oracle.com/export/java_re2/jdk/8u40/ea/b06/binaries/linux-x64/jre/lib/rt.jar] Thread 'DestroyJavaVM' has called transform() [Loaded java.lang.Shutdown$Lock from /net/scanas412.us.oracle.com/export/java_re2/jdk/8u40/ea/b06/binaries/linux-x64/jre/lib/rt.jar] result=1 ----------System.err:(14/906)---------- Exception in thread "main" java.lang.ClassCircularityError: sun/misc/URLClassPath$JarLoader$2 at sun.misc.URLClassPath$JarLoader.checkResource(URLClassPath.java:771) at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:843) at sun.misc.URLClassPath.getResource(URLClassPath.java:199) at java.net.URLClassLoader$1.run(URLClassLoader.java:364) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:360) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:344) at ParallelTransformerLoaderApp.loadClasses(ParallelTransformerLoaderApp.java:83) at ParallelTransformerLoaderApp.main(ParallelTransformerLoaderApp.java:45)
23-09-2014

Yumin, try adding trace options like -verbose:class (or other options more relevant to class instrumentation) to 8038468.sh to see how classes are being loaded/instrumented in the failure case. You can see the traces in the failed ParallelTransformerLoader.jtr file.
23-09-2014

Instead of just trying to reproduce the failure, need to analyze the test and VM source code to see how this could happen. Also, when the error happens, which classes exactly are detected as circularly dependent?
23-09-2014

Failure is reproducible again.
23-09-2014

Recent PITs result showed it is not reproducible. Closed.
19-09-2014

I run the following binaries against this test on Windows 7, cannot reproduce the failure: 1) The failed version: java version "1.9.0-ea-fastdebug" Java(TM) SE Runtime Environment (build 1.9.0-ea-fastdebug-b16) Java HotSpot(TM) Client VM (build 1.9.0-internal-201406061312.ctornqvi.hs-rt-fastdebug, mixed mode) 2) JDK1.9.0 b16: java version "1.9.0-ea" Java(TM) SE Runtime Environment (build 1.9.0-ea-b16) Java HotSpot(TM) Client VM (build 1.9.0-ea-b16, mixed mode) 3) Most recent JDK9 b25: java version "1.9.0-ea-fastdebug" Java(TM) SE Runtime Environment (build 1.9.0-ea-fastdebug-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.0-b62, mixed mode) Tested with both product and fastdebug versions. The failure I think is intermittent and could not repeat stable.
06-08-2014

TESTFAIL:java/lang/instrument/ParallelTransformerLoader.sh
11-04-2014