JDK-6392004 : Self-suspension holding Heap_lock blocks the use of FollowReferences in agent thread
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jvmti
  • Affected Version: 6
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • OS: generic
  • CPU: generic
  • Submitted: 2006-02-28
  • Updated: 2010-04-02
  • Resolved: 2006-03-20
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 6
6Resolved
Related Reports
Duplicate :  
Description
The attached script, runnbx.sh, runs NetBeans 4.1 under a simple debugger.
The debugger polls the keyboard for input. When 'Enter' is pressed,
the debugger calls JDI methods:
  - VirtualMachine.suspend() 
  - VirtualMachine.allClasses()
  - VirutalMachine.instanceCounts on all the classes

Rarely, a hang occurs here in the NetBeans process:

----------------- t@7 -----------------
0xff29f760	___lwp_mutex_lock + 0x8
0xfe8858cc	void Mutex::lock() + 0x144
0xfecc3f64	void JvmtiTagMap::follow_references(int,KlassHandle,_jobject*,const _jvmtiHeapCallbacks*,const void*) + 0xd0
0xfec8e4d0	jvmtiError JvmtiEnv::FollowReferences(int,_jclass*,_jobject*,const _jvmtiHeapCallbacks*,const void*) + 0x200
0xfec12008	jvmti_FollowReferences + 0x38c
0xfe7e3698	classInstanceCounts + 0x22c
0xfe7bc47c	instanceCounts + 0x12c
0xfe7c3960	debugLoop_run + 0x1bc
0xfe7da108	connectionInitiated + 0xf8
0xfe7da418	attachThread + 0x64
0xfecb2600	void JvmtiAgentThread::call_start_function() + 0xe0
0xfedca3d0	void JavaThread::thread_main_inner() + 0x38
0xfed3e2ec	java_start + 0x138
0xff3857b4	_lwp_start

See the attached file that contains the stack dumps for all threads.

I believe that this thread is waiting for the Heap_lock.

A full gcore dump is in  
  /net/jano.sfbay/export/disk20/serviceability/ws/regressionTestsInWaiting/core.followReferences
Finally reproduced this hang with Mustang-B75. I've attached
a thread dump for my hang as threads.log.17708. Here are the
two interesting threads:

THREAD t@9

This is our debugger threads waiting for a lock:

t@9(l@11) stopped in ___lwp_mutex_lock at 0xff29f990
0xff29f990: ___lwp_mutex_lock+0x0008:   ta       8
current thread: t@9
=>[1] ___lwp_mutex_lock(0x31200, 0xa, 0xff296ed4, 0x2b9f8, 0xff130000, 0xfe756d38), at 0xff29f990
  [2] Mutex::wait_for_lock_blocking_implementation(0x2c7c8, 0x104c00, 0x1ffc, 0x1, 0xfe72e000, 0x20980), at 0xfe1a6564
  [3] Mutex::lock(0x2c7c8, 0x104c00, 0xed5, 0x76ac0e, 0xfe752fdc, 0xfe72e000), at 0xfe0859ac
  [4] JvmtiTagMap::follow_references(0x1b8e680, 0x20, 0xed581774, 0x0, 0xed581878, 0xed5818b8), at 0xfe4c3ff4
  [5] JvmtiEnv::FollowReferences(0x1c593f0, 0x20, 0x36c, 0x0, 0xed581878, 0xed5818b8), at 0xfe48e540
  [6] jvmti_FollowReferences(0x1c593f0, 0x20, 0x0, 0x0, 0xed581878, 0xed5818b8), at 0xfe412078
  [7] classInstanceCounts(0xfe7face0, 0x1c593f0, 0x0, 0x0, 0xfe7face0, 0x0), at 0xfe7e3698
  [8] instanceCounts(0xed581b00, 0xed5819a0, 0x1dda370, 0x104ce4, 0xfe7fa714, 0x400), at 0xfe7bc47c
  [9] debugLoop_run(0x400, 0x0, 0xfe7fa714, 0x1, 0xfe7face0, 0xfe7bc350), at 0xfe7c3960
  [10] connectionInitiated(0xfde22bf0, 0xa4c, 0x1, 0xfe7fa714, 0xfe7fb160, 0x800), at 0xfe7da108
  [11] attachThread(0xfe7fac68, 0x554, 0xfde22bf0, 0xfe7fa714, 0x2035c, 0x400), at 0xfe7da418
  [12] JvmtiAgentThread::call_start_function(0x104c00, 0x29714, 0x2b800, 0xfe72e000, 0x29400, 0x7), at 0xfe4b2694
  [13] JavaThread::thread_main_inner(0x104c00, 0xed825ec0, 0x1043a8, 0xfe72e000, 0x11f018, 0x0), at 0xfe5ca358
  [14] java_start(0x104c00, 0x5, 0x452c, 0xfe72e000, 0xfe6a3b89, 0x29400), at 0xfe53e374

Several other threads are self-suspend, but this one is interesting:

THREAD t@26

This thread has self-suspended while waiting for a VM operation to
complete. I'm guessing since there is a GenCollectorPolicy::mem_allocate_work()
call on the stack, that this thread is holding the Heap_lock.

t@26(l@28) stopped in ___lwp_cond_wait at 0xff29fa08
0xff29fa08: ___lwp_cond_wait+0x0004:    ta       8
current thread: t@26
=>[1] ___lwp_cond_wait(0x4e0880, 0x4e0868, 0x0, 0x0, 0x0, 0x0), at 0xff29fa08
  [2] _lwp_cond_wait(0x4e0880, 0x4e0868, 0xfe767630, 0x1406f40, 0x0, 0x1), at 0xff296f4c
  [3] Monitor::wait(0x6516b0, 0x4e0400, 0x0, 0xfe72e000, 0x4e0840, 0x0), at 0xfe52e04c
  [4] os::pd_self_suspend_thread(0x4e0400, 0x28e40, 0xec0, 0x1, 0x80000000, 0xfe72e000), at 0xfe53eb5c
  [5] JavaThread::java_suspend_self(0x40000000, 0x1, 0x4e0400, 0x1406f40, 0x0, 0xdeac), at 0xfe5cbea4
  [6] Monitor::wait(0x30e78, 0x4e0400, 0x4e09e0, 0xfe72e000, 0x30ea8, 0x4e0400), at 0xfe52e668
  [7] VMThread::execute(0xec000b1c, 0x35c00, 0xfe75afe8, 0x4e0400, 0x1, 0xfe72e000), at 0xfe1797cc
  [8] GenCollectorPolicy::mem_allocate_work(0x34ec8, 0x804, 0x67, 0xec000bd3, 0xec000b58, 0xfe72e000), at 0xfe2d7ab0
  [9] instanceKlass::allocate_objArray(0xfe340954, 0x69d7d0, 0x800, 0x804, 0x4e0400, 0x2d93c), at 0xfe0982c0
  [10] InterpreterRuntime::anewarray(0x4e0400, 0xf7761d70, 0x8, 0x800, 0x6, 0xfe72e000), at 0xfe098a5c
  [11] 0xfbc2065c(0xee4eb200, 0xb7, 0xf78c9cd0, 0xfbc1f2a0, 0x71a, 0xec000cc0), at 0xfbc2065b
  [12] 0xfbc05a30(0xee4eb200, 0xf124bbb8, 0xf78c9c78, 0xfbc1f9e8, 0x86d, 0xec000d60), at 0xfbc05a2f
  [13] 0xfbe2d43c(0xf0f86bf8, 0xf124bbb8, 0xee530010, 0x1, 0x974, 0xec000e30), at 0xfbe2d43b
  [14] 0xfbc058c0(0xf0f8fa58, 0xb6, 0xf78a4c24, 0xfbc1c5c8, 0xa1, 0xec000ef8), at 0xfbc058bf
  [15] 0xfbc058c0(0xf0f86850, 0xb7, 0xf786f2b8, 0xfbc1f6c0, 0xad, 0xec000fb8), at 0xfbc058bf
  [16] 0xfbc05a30(0xf0f86850, 0xb7, 0xf786ef60, 0xfbc1f9a0, 0xac, 0xec001078), at 0xfbc05a2f
  [17] 0xfbc05a30(0xf0f86850, 0xb7, 0xf786ef60, 0xfbc1f9a0, 0xa7, 0xec001138), at 0xfbc05a2f
  [18] 0xfbc05a30(0xf0f86850, 0xb7, 0xf786ef60, 0xfbc1f9a0, 0x6e, 0xec0011f8), at 0xfbc05a2f
  [19] 0xfbc05a30(0xf0f86850, 0xb7, 0xf786ef60, 0xfbc1f9a0, 0xedd349e0, 0xec0012b8), at 0xfbc05a2f
  [20] 0xfbc05a30(0xedcbf498, 0xb6, 0x0, 0xfbc1f9f0, 0xedcc38a0, 0xec001378), at 0xfbc05a2f
  [21] 0xfbc058c0(0xf06a7f40, 0xb6, 0x0, 0xfbc1f9a0, 0xf06a7f40, 0xec001450), at 0xfbc058bf
  [22] 0xfbc058c0(0xf06a7f40, 0xb7, 0x0, 0xfbc1c5c8, 0xf6773ad8, 0xec001520), at 0xfbc058bf
  [23] 0xfbc05a30(0xf06a7f40, 0x80000000, 0x0, 0xfbc1f9e8, 0x29714, 0xec0015d0), at 0xfbc05a2f
  [24] 0xfbc05a30(0xeda19f20, 0xf582ea50, 0x0, 0xfbc1f710, 0x6, 0xec001658), at 0xfbc05a2f
  [25] 0xfbc05f50(0xeda19f48, 0xb7, 0x0, 0xfbc1fd90, 0xf06b0120, 0xec0016d8), at 0xfbc05f4f
  [26] 0xfbc05a30(0xeda19f48, 0xb6, 0x0, 0xfbc1f9a0, 0x80000000, 0xec001758), at 0xfbc05a2f
  [27] 0xfbc05a30(0xec001d38, 0xfe756d38, 0x0, 0xfbc1f6c0, 0xee076128, 0xec001800), at 0xfbc05a2f
  [28] 0xfbc0021c(0xec001908, 0xec001bb0, 0xa, 0xf61ffdd8, 0xfbc0f700, 0xec001a98), at 0xfbc0021b
  [29] JavaCalls::call_helper(0x1, 0xf61ffdd8, 0xec001a90, 0x4e0400, 0x69d7b0, 0xec0018e8), at 0xfe0dd270
  [30] JavaCalls::call_virtual(0xec001ba8, 0x69d7b4, 0xfe767f88, 0x69d7c0, 0xec001a90, 0x4e0400), at 0xfe378f10
  [31] JavaCalls::call_virtual(0xec001ba8, 0xec001ba4, 0xec001ba0, 0xec001b9c, 0xec001b98, 0x69d7b4), at 0xfe1884b4
  [32] thread_entry(0xf58320b8, 0x4e0400, 0x39800, 0xfe768068, 0xfe767f88, 0xfe767a08), at 0xfe19b648
  [33] JavaThread::thread_main_inner(0x4e0400, 0xee076128, 0x59eaf8, 0xfe72e000, 0x4ddc40, 0x0), at 0xfe5ca358
  [34] java_start(0x4e0400, 0x16, 0x452c, 0xfe72e000, 0xfe6a3b89, 0x29400), at 0xfe53e374

Comments
EVALUATION Jim has agreed that this is a duplicate of 6321448. Closing as such...
20-03-2006

EVALUATION Below is the key fragment from thread t@26: [4] os::pd_self_suspend_thread(0x4e0400, 0x28e40, 0xec0, 0x1, 0x80000000, 0xfe72e000), at 0xfe53eb5c [5] JavaThread::java_suspend_self(0x40000000, 0x1, 0x4e0400, 0x1406f40, 0x0, 0xdeac), at 0xfe5cbea4 [6] Monitor::wait(0x30e78, 0x4e0400, 0x4e09e0, 0xfe72e000, 0x30ea8, 0x4e0400), at 0xfe52e668 [7] VMThread::execute(0xec000b1c, 0x35c00, 0xfe75afe8, 0x4e0400, 0x1, 0xfe72e000), at 0xfe1797cc [8] GenCollectorPolicy::mem_allocate_work(0x34ec8, 0x804, 0x67, 0xec000bd3, 0xec000b58, 0xfe72e000), at 0xfe2d7ab0 [9] instanceKlass::allocate_objArray(0xfe340954, 0x69d7d0, 0x800, 0x804, 0x4e0400, 0x2d93c), at 0xfe0982c0 [10] InterpreterRuntime::anewarray(0x4e0400, 0xf7761d70, 0x8, 0x800, 0x6, 0xfe72e000), at 0xfe098a5c [11] 0xfbc2065c(0xee4eb200, 0xb7, 0xf78c9cd0, 0xfbc1f2a0, 0x71a, 0xec000cc0), at 0xfbc2065b This thread is trying to allocate memory and has to make a VM operation call to do so. All of the GC related VM operations are subclassed off of VM_GC_Operation. In the prologue for VM_GC_Operation, the Heap_lock is grabbed: bool VM_GC_Operation::doit_prologue() { assert(Thread::current()->is_Java_thread(), "just checking"); acquire_pending_list_lock(); // If the GC count has changed someone beat us to the collection // Get the Heap_lock after the pending_list_lock. Heap_lock->lock(); // Check invocations This means that when the Monitor::wait() call in VMThread::execute() detects an external suspend request and self-suspends, then this thread has self-suspended while holding the Heap_lock. Since the thread (t@9) that caused this thread to be externally suspended is blocked waiting on the Heap_lock, we have a classic deadlock. This is indeed a variation of the following bug: 6321448 3/4 self-suspension with JVM-internal monitors and mutexes may not be needed I'm planning to close this bug as a duplicate of 6321448.
17-03-2006