The attached script, runnbx.sh, runs NetBeans 4.1 under a simple debugger.
The debugger polls the keyboard for input. When 'Enter' is pressed,
the debugger calls JDI methods:
- VirtualMachine.suspend()
- VirtualMachine.allClasses()
- VirutalMachine.instanceCounts on all the classes
Rarely, a hang occurs here in the NetBeans process:
----------------- t@7 -----------------
0xff29f760 ___lwp_mutex_lock + 0x8
0xfe8858cc void Mutex::lock() + 0x144
0xfecc3f64 void JvmtiTagMap::follow_references(int,KlassHandle,_jobject*,const _jvmtiHeapCallbacks*,const void*) + 0xd0
0xfec8e4d0 jvmtiError JvmtiEnv::FollowReferences(int,_jclass*,_jobject*,const _jvmtiHeapCallbacks*,const void*) + 0x200
0xfec12008 jvmti_FollowReferences + 0x38c
0xfe7e3698 classInstanceCounts + 0x22c
0xfe7bc47c instanceCounts + 0x12c
0xfe7c3960 debugLoop_run + 0x1bc
0xfe7da108 connectionInitiated + 0xf8
0xfe7da418 attachThread + 0x64
0xfecb2600 void JvmtiAgentThread::call_start_function() + 0xe0
0xfedca3d0 void JavaThread::thread_main_inner() + 0x38
0xfed3e2ec java_start + 0x138
0xff3857b4 _lwp_start
See the attached file that contains the stack dumps for all threads.
I believe that this thread is waiting for the Heap_lock.
A full gcore dump is in
/net/jano.sfbay/export/disk20/serviceability/ws/regressionTestsInWaiting/core.followReferences
Finally reproduced this hang with Mustang-B75. I've attached
a thread dump for my hang as threads.log.17708. Here are the
two interesting threads:
THREAD t@9
This is our debugger threads waiting for a lock:
t@9(l@11) stopped in ___lwp_mutex_lock at 0xff29f990
0xff29f990: ___lwp_mutex_lock+0x0008: ta 8
current thread: t@9
=>[1] ___lwp_mutex_lock(0x31200, 0xa, 0xff296ed4, 0x2b9f8, 0xff130000, 0xfe756d38), at 0xff29f990
[2] Mutex::wait_for_lock_blocking_implementation(0x2c7c8, 0x104c00, 0x1ffc, 0x1, 0xfe72e000, 0x20980), at 0xfe1a6564
[3] Mutex::lock(0x2c7c8, 0x104c00, 0xed5, 0x76ac0e, 0xfe752fdc, 0xfe72e000), at 0xfe0859ac
[4] JvmtiTagMap::follow_references(0x1b8e680, 0x20, 0xed581774, 0x0, 0xed581878, 0xed5818b8), at 0xfe4c3ff4
[5] JvmtiEnv::FollowReferences(0x1c593f0, 0x20, 0x36c, 0x0, 0xed581878, 0xed5818b8), at 0xfe48e540
[6] jvmti_FollowReferences(0x1c593f0, 0x20, 0x0, 0x0, 0xed581878, 0xed5818b8), at 0xfe412078
[7] classInstanceCounts(0xfe7face0, 0x1c593f0, 0x0, 0x0, 0xfe7face0, 0x0), at 0xfe7e3698
[8] instanceCounts(0xed581b00, 0xed5819a0, 0x1dda370, 0x104ce4, 0xfe7fa714, 0x400), at 0xfe7bc47c
[9] debugLoop_run(0x400, 0x0, 0xfe7fa714, 0x1, 0xfe7face0, 0xfe7bc350), at 0xfe7c3960
[10] connectionInitiated(0xfde22bf0, 0xa4c, 0x1, 0xfe7fa714, 0xfe7fb160, 0x800), at 0xfe7da108
[11] attachThread(0xfe7fac68, 0x554, 0xfde22bf0, 0xfe7fa714, 0x2035c, 0x400), at 0xfe7da418
[12] JvmtiAgentThread::call_start_function(0x104c00, 0x29714, 0x2b800, 0xfe72e000, 0x29400, 0x7), at 0xfe4b2694
[13] JavaThread::thread_main_inner(0x104c00, 0xed825ec0, 0x1043a8, 0xfe72e000, 0x11f018, 0x0), at 0xfe5ca358
[14] java_start(0x104c00, 0x5, 0x452c, 0xfe72e000, 0xfe6a3b89, 0x29400), at 0xfe53e374
Several other threads are self-suspend, but this one is interesting:
THREAD t@26
This thread has self-suspended while waiting for a VM operation to
complete. I'm guessing since there is a GenCollectorPolicy::mem_allocate_work()
call on the stack, that this thread is holding the Heap_lock.
t@26(l@28) stopped in ___lwp_cond_wait at 0xff29fa08
0xff29fa08: ___lwp_cond_wait+0x0004: ta 8
current thread: t@26
=>[1] ___lwp_cond_wait(0x4e0880, 0x4e0868, 0x0, 0x0, 0x0, 0x0), at 0xff29fa08
[2] _lwp_cond_wait(0x4e0880, 0x4e0868, 0xfe767630, 0x1406f40, 0x0, 0x1), at 0xff296f4c
[3] Monitor::wait(0x6516b0, 0x4e0400, 0x0, 0xfe72e000, 0x4e0840, 0x0), at 0xfe52e04c
[4] os::pd_self_suspend_thread(0x4e0400, 0x28e40, 0xec0, 0x1, 0x80000000, 0xfe72e000), at 0xfe53eb5c
[5] JavaThread::java_suspend_self(0x40000000, 0x1, 0x4e0400, 0x1406f40, 0x0, 0xdeac), at 0xfe5cbea4
[6] Monitor::wait(0x30e78, 0x4e0400, 0x4e09e0, 0xfe72e000, 0x30ea8, 0x4e0400), at 0xfe52e668
[7] VMThread::execute(0xec000b1c, 0x35c00, 0xfe75afe8, 0x4e0400, 0x1, 0xfe72e000), at 0xfe1797cc
[8] GenCollectorPolicy::mem_allocate_work(0x34ec8, 0x804, 0x67, 0xec000bd3, 0xec000b58, 0xfe72e000), at 0xfe2d7ab0
[9] instanceKlass::allocate_objArray(0xfe340954, 0x69d7d0, 0x800, 0x804, 0x4e0400, 0x2d93c), at 0xfe0982c0
[10] InterpreterRuntime::anewarray(0x4e0400, 0xf7761d70, 0x8, 0x800, 0x6, 0xfe72e000), at 0xfe098a5c
[11] 0xfbc2065c(0xee4eb200, 0xb7, 0xf78c9cd0, 0xfbc1f2a0, 0x71a, 0xec000cc0), at 0xfbc2065b
[12] 0xfbc05a30(0xee4eb200, 0xf124bbb8, 0xf78c9c78, 0xfbc1f9e8, 0x86d, 0xec000d60), at 0xfbc05a2f
[13] 0xfbe2d43c(0xf0f86bf8, 0xf124bbb8, 0xee530010, 0x1, 0x974, 0xec000e30), at 0xfbe2d43b
[14] 0xfbc058c0(0xf0f8fa58, 0xb6, 0xf78a4c24, 0xfbc1c5c8, 0xa1, 0xec000ef8), at 0xfbc058bf
[15] 0xfbc058c0(0xf0f86850, 0xb7, 0xf786f2b8, 0xfbc1f6c0, 0xad, 0xec000fb8), at 0xfbc058bf
[16] 0xfbc05a30(0xf0f86850, 0xb7, 0xf786ef60, 0xfbc1f9a0, 0xac, 0xec001078), at 0xfbc05a2f
[17] 0xfbc05a30(0xf0f86850, 0xb7, 0xf786ef60, 0xfbc1f9a0, 0xa7, 0xec001138), at 0xfbc05a2f
[18] 0xfbc05a30(0xf0f86850, 0xb7, 0xf786ef60, 0xfbc1f9a0, 0x6e, 0xec0011f8), at 0xfbc05a2f
[19] 0xfbc05a30(0xf0f86850, 0xb7, 0xf786ef60, 0xfbc1f9a0, 0xedd349e0, 0xec0012b8), at 0xfbc05a2f
[20] 0xfbc05a30(0xedcbf498, 0xb6, 0x0, 0xfbc1f9f0, 0xedcc38a0, 0xec001378), at 0xfbc05a2f
[21] 0xfbc058c0(0xf06a7f40, 0xb6, 0x0, 0xfbc1f9a0, 0xf06a7f40, 0xec001450), at 0xfbc058bf
[22] 0xfbc058c0(0xf06a7f40, 0xb7, 0x0, 0xfbc1c5c8, 0xf6773ad8, 0xec001520), at 0xfbc058bf
[23] 0xfbc05a30(0xf06a7f40, 0x80000000, 0x0, 0xfbc1f9e8, 0x29714, 0xec0015d0), at 0xfbc05a2f
[24] 0xfbc05a30(0xeda19f20, 0xf582ea50, 0x0, 0xfbc1f710, 0x6, 0xec001658), at 0xfbc05a2f
[25] 0xfbc05f50(0xeda19f48, 0xb7, 0x0, 0xfbc1fd90, 0xf06b0120, 0xec0016d8), at 0xfbc05f4f
[26] 0xfbc05a30(0xeda19f48, 0xb6, 0x0, 0xfbc1f9a0, 0x80000000, 0xec001758), at 0xfbc05a2f
[27] 0xfbc05a30(0xec001d38, 0xfe756d38, 0x0, 0xfbc1f6c0, 0xee076128, 0xec001800), at 0xfbc05a2f
[28] 0xfbc0021c(0xec001908, 0xec001bb0, 0xa, 0xf61ffdd8, 0xfbc0f700, 0xec001a98), at 0xfbc0021b
[29] JavaCalls::call_helper(0x1, 0xf61ffdd8, 0xec001a90, 0x4e0400, 0x69d7b0, 0xec0018e8), at 0xfe0dd270
[30] JavaCalls::call_virtual(0xec001ba8, 0x69d7b4, 0xfe767f88, 0x69d7c0, 0xec001a90, 0x4e0400), at 0xfe378f10
[31] JavaCalls::call_virtual(0xec001ba8, 0xec001ba4, 0xec001ba0, 0xec001b9c, 0xec001b98, 0x69d7b4), at 0xfe1884b4
[32] thread_entry(0xf58320b8, 0x4e0400, 0x39800, 0xfe768068, 0xfe767f88, 0xfe767a08), at 0xfe19b648
[33] JavaThread::thread_main_inner(0x4e0400, 0xee076128, 0x59eaf8, 0xfe72e000, 0x4ddc40, 0x0), at 0xfe5ca358
[34] java_start(0x4e0400, 0x16, 0x452c, 0xfe72e000, 0xfe6a3b89, 0x29400), at 0xfe53e374