JDK-7071060 : Deadlock occurs in java.lang.instrumentation
  • Type: Bug
  • Component: core-svc
  • Sub-Component: java.lang.instrument
  • Affected Version: 5.0u31,7
  • Priority: P4
  • Status: Closed
  • Resolution: Won't Fix
  • OS: linux_redhat_5.0
  • CPU: x86
  • Submitted: 2011-07-26
  • Updated: 2023-12-14
  • Resolved: 2013-10-10
Related Reports
Duplicate :  
JDK-7071061 - Deadlock occurs with URLClassLoader
Relates :  
JDK-8258800 - Deprecate -XX:+AlwaysLockClassLoader
Relates :  
JDK-5088398 - java.lang.instrument TCK test deadlock (test11)
Relates :  
JDK-6835233 - Fedora 9 jdk regression test failed: java/lang/instrument/ParallelTransformerLoader.sh
Description
A CU faces with a serious deadlock in java.lang.instrumentation and stop their system.

CONFIGURATION:
OS :
xxxx 116 % uname -a
Linux xxxx 2.6.18-164.11.1.el5 #1 SMP Wed Jan 6 13:26:04 EST 2010 x86
_64 x86_64 x86_64 GNU/Linux

xxxx 117 % more /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)

JDK : JDK5u31(32bit)/JDK5u30(32bit)
java version "1.5.0_31"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_31-b02)
Java HotSpot(TM) Server VM (build 1.5.0_31-b02, mixed mode)

java version "1.5.0_30"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_30-b03)
Java HotSpot(TM) Server VM (build 1.5.0_30-b03, mixed mode)

REPRODUCE:
1. unfold attached TP1.zip
2. Goto TP1 directory
3. Launch
    "java -javaagent:myagent.jar test1"

You will see the following message.

<premain>
transform:StartInPremain(sun.misc.Launcher$AppClassLoader@18e3e60)
transform:sun/misc/URLClassPath$FileLoader$1(null)
transform:test1(sun.misc.Launcher$AppClassLoader@18e3e60)
transform:LoadBySystemCL(sun.misc.Launcher$AppClassLoader@18e3e60)
transform:javax/management/MBeanInfo(null)

4. Open another console, find out the processID, and launch
   "kill -3 <processID>"

You will see an deadlock in the thread dump.
(ex. please see the attached deadlock-log-jdk5u31.txt. )

INVESTIGATION:

An deadlock occurs between "bootstrap class loader" and a lock(synchronized)
gotten in javaagent.(This can be confirmed in thread dump)
When a user use instrument functionality, the user implements transform() method of java.lang.instrument.ClassFileTransformer.
If the target classes of classloading are classes in rt.jar,  
transform() is called under the status of which "bootstrap class loader" is locked.
This seems to result in the deadlock.
If this is a customer with a a support contract this must go through customer support to get the correct prio. thanks!
Comments
EVALUATION Note the deadlock information: "Thread-0": waiting to lock monitor 0x0811d724 (object 0xa0800000, a [[I), seems to report the incorrect type for the object being locked. It is not a [[I but the ClassLoader instance.
27-07-2011
EVALUATION There is a deadlock between the agent code and the classloading framework: Found one Java-level deadlock: ============================= "Thread-0": waiting to lock monitor 0x08cfa5cc (object 0xd0b14a48, a [[I), which is held by "main" "main": waiting to lock monitor 0x08cfa58c (object 0xd0c80bf8, a java.lang.Class), which is held by "Thread-0" Java stack information for the threads listed above: =================================================== "Thread-0": at java.lang.ClassLoader.findBootstrapClass(Native Method) at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:892) at java.lang.ClassLoader.loadClass(ClassLoader.java:302) - locked <0xf0b28018> (a sun.misc.Launcher$ExtClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:300) - locked <0xf0b2fdd0> (a sun.misc.Launcher$AppClassLoader) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) - locked <0xf0b2fdd0> (a sun.misc.Launcher$AppClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) - locked <0xf0b2fdd0> (a sun.misc.Launcher$AppClassLoader) at MyAgent.transform(MyAgent.java:19) - locked <0xd0c80bf8> (a java.lang.Class) at sun.instrument.TransformerManager.transform(TransformerManager.java:1 22) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java :155) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:621) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:12 4) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) - locked <0xf0b2fdd0> (a sun.misc.Launcher$AppClassLoader) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) - locked <0xf0b2fdd0> (a sun.misc.Launcher$AppClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) - locked <0xf0b2fdd0> (a sun.misc.Launcher$AppClassLoader) at StartInPremain.run(MyAgent.java:33) "main": at MyAgent.transform(MyAgent.java:17) - waiting to lock <0xd0c80bf8> (a java.lang.Class) at sun.instrument.TransformerManager.transform(TransformerManager.java:1 22) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java :155) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at test1.main(test1.java:9) Found 1 deadlock. The agent code acquires the lock of the MyAgent class and while holding that lock performs an action that involves classloading. The classloading in turn tries to use the agent. Both the main thread and the agent thread are attempting these actions and so we get a classic deadlock. As discussed in 7071061 Agent writers need to understand how their code is executed within the classloading process and so be aware of any explicit locking they perform while doing classloading themselves. The general advice when calling out to external code (ie when the agent calls into the classloading framework) is to make "open calls" - that is to call the external code without holding any locks (ref: "Concurrent Programming in Java" by Doug Leal, "Java Concurrency in Practice" by Brian Goetz et al.)
27-07-2011