JDK-8153711 : [REDO] JDWP: Memory Leak: GlobalRefs never deleted when processing invokeMethod command
  • Type: Bug
  • Component: core-svc
  • Sub-Component: debugger
  • Affected Version: 9
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2016-04-07
  • Updated: 2022-01-03
  • Resolved: 2016-09-15
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 8 JDK 9
8u152Fixed 9 b139Fixed
Related Reports
Blocks :  
JDK-8156500 - Move Reference pending list into VM to prevent deadlocks
Relates :  
JDK-8176567 - nsk/jdi/ReferenceType/instances/instances002: TestFailure: Unexpected size of referenceType.instances(nsk.share.jdi.TestInterfaceImplementer1): 11, expected: 10
Relates :  
JDK-8152586 - com/sun/jdi/InterfaceMethodsTest.java fails in hs nightly
Relates :  
JDK-8154529 - some places in the invoke.c that use InvokeRequest* not protected with invokerLock
Relates :  
JDK-8181419 - Race in jdwp invoker handling may lead to crashes or invalid results
Relates :  
JDK-8156498 - more places in the invoke.c that need protection with the invokerLock
Relates :  
JDK-8152686 - com/sun/jdi/OomDebugTest.java fails with java.lang.reflect.InvocationTargetException
Relates :  
JDK-8156500 - Move Reference pending list into VM to prevent deadlocks
Relates :  
JDK-8152985 - nsk/jdi tests fail with unexpected com.sun.jdi.InvocationException
Relates :  
JDK-4858370 - JDWP: Memory Leak: GlobalRefs never deleted when processing invokeMethod command
Relates :  
JDK-8153673 - [BACKOUT] JDWP: Memory Leak: GlobalRefs never deleted when processing invokeMethod command
Description
The fix of:
  JDK-4858370 JDWP: Memory Leak: GlobalRefs never deleted when processing invokeMethod command

was backed out with:
  JDK-8153673 [BACKOUT] JDWP: Memory Leak: GlobalRefs never deleted when processing invokeMethod command

because of the following regressions:
  JDK-8152686 com/sun/jdi/OomDebugTest.java fails with java.lang.reflect.InvocationTargetException
  JDK-8152586 com/sun/jdi/InterfaceMethodsTest.java fails in hs nightly
  JDK-8152985 nsk/jdi tests fail with unexpected com.sun.jdi.InvocationException

A corrected fix for the original problem with GlobalRefs memory leak is needed.
Comments
URL: http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/4c843eb35b8a User: lana Date: 2016-10-05 20:01:48 +0000
05-10-2016
URL: http://hg.openjdk.java.net/jdk9/hs/jdk/rev/4c843eb35b8a User: dsamersoff Date: 2016-09-15 11:09:47 +0000
15-09-2016
Posted for review: http://mail.openjdk.java.net/pipermail/serviceability-dev/2016-September/020407.html
09-09-2016
The NPE problem in InvokeTest and the triggered assertion in OomDebugTest is caused by clearing the request's instance and clazz members *after* the IO operation. Like every other failures I've seen for this bug it's not reliably reproducible. Especially not when debugging this itself. It maybe related to the sequence of monitor_exit->perform IO->monitor_enter->toss references. Perhaps there is a schedule where the response has been sent back, the next invoke starts for the same app thread and it is just far enough along so that the tossing of the references becomes observable by the next request. I really don't know. However, testing showed that moving this up prevents this assertion from being triggered. Thus, I'm now clearing global references - the ones we can clear before sending back the response to the client - in the same block while still holding the invoker and event handler locks as the rest of the operations being done in completeInvokeRequest. Once the response has been sent to the client there are still two global references to clear: The one for the return value and a possible exception which might have occurred. Since they are required for sending the response we do this after it's been sent.
09-09-2016
Latest webrev which has the NPE problem on solaris x86_64 fixed. Turns out, getting OomDebugTest working reliably on solaris also fixes the InvokeTest problem. I need to do some more testing, but it's looking good so far: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8153711/webrev.03/
08-09-2016
Tried and failed to reproduce, so I went back through old test reports. Looks like I mis-remembered, and it was only once. And looking more closely, it turned out to be an ephemeral test reporting artifact. So not a problem after all. Sorry for the noise.
02-09-2016
No, sorry. I hadn't investigated further, since I was focusing on JDK-8156500. I'll try to reproduce the problem, but it might take a few days.
01-09-2016
@Kim: Any details about the failures you were getting on SPARC. I don't have a SPARC machine available myself.
01-09-2016
With JDK-8156500 fixed, the proposed OomDebugTest now runs quite reliably for me. However, while testing that change + the proposed webrev.02 I saw frequent / consistent failures on SPARC for ./test/com/sun/jdi/InvokeTest.java, which weren't reproducible with only the fix for JDK-8156500.
01-09-2016
@David: It's been on the back-burner for a while, but I'll work on this again this or next week. I don't have an ETA for proper resolution yet, though.
24-08-2016
Severin: are you still actively working on this issue? As we approach Rampdown Phase 1 (RDP1) and head towards Zero-bug-bounce (ZBB) we need to track all open issues. Thanks.
24-08-2016
OomDebugTest also fails on this system. One failure is the result of a crash in OomDebugTestTarget: DEBUG: Done invoking method via debugger. # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/jniHandles.hpp:197 # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (/export/home/sgehwolf/Documents/openjdk9-hs/hotspot/src/share/vm/runtime/jniHandles.hpp:197), pid=20440, tid=2 # assert(handle != 0L) failed: JNI handle should not be null # # JRE version: OpenJDK Runtime Environment (9.0) (fastdebug build 9-internal+0-2016-05-17-185224.sgehwolf.openjdk9-hs) # Java VM: OpenJDK 64-Bit Server VM (fastdebug 9-internal+0-2016-05-17-185224.sgehwolf.openjdk9-hs, mixed mode, tiered, compressed oops, g1 gc, solaris-amd64) # Core dump will be written. Default location: /export/home/sgehwolf/Documents/openjdk9-hs/JTwork/scratch/core or core.20440 # # An error report file with more information is saved as: # /export/home/sgehwolf/Documents/openjdk9-hs/JTwork/scratch/hs_err_pid20440.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The corresponding stack trace from hs_err.log is: V [libjvm.so+0x17b6b94] oop JNIHandles::resolve_non_null(_jobject*)+0x54 V [libjvm.so+0x1d8e766] instanceOop alloc_object(_jclass*,Thread*)+0x26 V [libjvm.so+0x1d8f1d2] jni_NewObjectA+0x252 C [libjdwp.so+0x3a130] invokeConstructor+0x70 C [libjdwp.so+0x3b3e3] invoker_doInvoke+0x113 C [libjdwp.so+0x32250] reportEvents+0x130 C [libjdwp.so+0x32804] event_callback+0x324 C [libjdwp.so+0x32e75] cbBreakpoint+0x145 V [libjvm.so+0x1f8c026] void JvmtiExport::post_raw_breakpoint(JavaThread*,Method*,unsigned char*)+0x996 V [libjvm.so+0x1d2aa8b] void InterpreterRuntime::_breakpoint(JavaThread*,Method*,unsigned char*)+0x19b j OomDebugTestTarget.entry()V+0 j OomDebugTestTarget.main([Ljava/lang/String;)V+15
19-05-2016
Status update: I was able to reproduce this locally.
19-05-2016
Thanks. I'll see if I can find a solaris x64 box and will try to reproduce.
09-05-2016
The jprt push of the fix failed with the NullPointerException: TEST: com/sun/jdi/InvokeTest.java TEST JDK: /opt/jprt/T/P1/100525.sspitsyn/testproduct/solaris_x64_5.11-fastdebug ACTION: build -- Passed. All files up to date REASON: User specified action: run build TestScaffold VMConnection TargetListener TargetAdapter TIME: 0.0 seconds messages: command: build TestScaffold VMConnection TargetListener TargetAdapter reason: User specified action: run build TestScaffold VMConnection TargetListener TargetAdapter elapsed time (seconds): 0.0 ACTION: compile -- Passed. Compilation successful REASON: User specified action: run compile -g InvokeTest.java TIME: 0.106 seconds messages: command: compile -g /opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi/InvokeTest.java reason: User specified action: run compile -g InvokeTest.java Mode: agentvm elapsed time (seconds): 0.106 configuration: Boot Layer (javac execution environment) class path: /opt/jprt/jib-data/install/java/re/jtreg/4.2/promoted/all/b02/bundles/jtreg_bin-4.2.zip/jtreg/lib/javatest.jar /opt/jprt/jib-data/install/java/re/jtreg/4.2/promoted/all/b02/bundles/jtreg_bin-4.2.zip/jtreg/lib/jtreg.jar patch: java.base /opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/patches/java.base javac compilation environment class path: /opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi /opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi /opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun rerun: DISPLAY=scaaa563.us.oracle.com:514 \ HOME=/opt/jprt/jprtadm \ JDK8_HOME= \ LANG=C \ LC_ALL=C \ LC_CTYPE= \ LD_LIBRARY_PATH=/opt/jprt/T/P1/100525.sspitsyn/bundles/testBundle/jdk/jtreg/native \ PATH=/bin:/usr/bin \ TZ=localtime \ /opt/jprt/T/P1/100525.sspitsyn/testproduct/solaris_x64_5.11-fastdebug/bin/javac \ -J-ea \ -J-esa \ -J-d64 \ -J-server \ -J-Xmx512m \ -J-Duser.home=/opt/jprt/T/P1/100525.sspitsyn \ -J-Djava.io.tmpdir=/opt/jprt/T/P1/100525.sspitsyn/io/solaris_x64_5.11-fastdebug-c2-jdk_svc_sanity \ -J-d64 \ -J-server \ -J-Djava.library.path=/opt/jprt/T/P1/100525.sspitsyn/bundles/testBundle/jdk/jtreg/native \ -J-Dtest.class.path.prefix=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun \ -J-Dtest.src=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi \ -J-Dtest.src.path=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun \ -J-Dtest.classes=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi \ -J-Dtest.class.path=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun \ -J-Dtest.vm.opts='-ea -esa -d64 -server -Xmx512m' \ -J-Dtest.tool.vm.opts='-J-ea -J-esa -J-d64 -J-server -J-Xmx512m' \ -J-Dtest.compiler.opts= \ -J-Dtest.java.opts='-Duser.home=/opt/jprt/T/P1/100525.sspitsyn -Djava.io.tmpdir=/opt/jprt/T/P1/100525.sspitsyn/io/solaris_x64_5.11-fastdebug-c2-jdk_svc_sanity -d64 -server' \ -J-Dtest.jdk=/opt/jprt/T/P1/100525.sspitsyn/testproduct/solaris_x64_5.11-fastdebug \ -J-Dcompile.jdk=/opt/jprt/T/P1/100525.sspitsyn/testproduct/solaris_x64_5.11-fastdebug \ -J-Dtest.timeout.factor=4.0 \ -J-Dtest.modules=jdk.jdi \ -J-Dtest.nativepath=/opt/jprt/T/P1/100525.sspitsyn/bundles/testBundle/jdk/jtreg/native \ -d /opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi \ -sourcepath /opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun \ -classpath /opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun \ -g /opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi/InvokeTest.java direct: Note: /opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi/InvokeTest.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. ACTION: build -- Passed. All files up to date REASON: Named class compiled on demand TIME: 0.0 seconds messages: command: build InvokeTest reason: Named class compiled on demand elapsed time (seconds): 0.0 ACTION: driver -- Failed. Execution failed: `main' threw exception: java.lang.NullPointerException REASON: User specified action: run driver InvokeTest TIME: 0.541 seconds messages: command: driver InvokeTest reason: User specified action: run driver InvokeTest Mode: agentvm elapsed time (seconds): 0.541 configuration: Boot Layer class path: /opt/jprt/jib-data/install/java/re/jtreg/4.2/promoted/all/b02/bundles/jtreg_bin-4.2.zip/jtreg/lib/javatest.jar /opt/jprt/jib-data/install/java/re/jtreg/4.2/promoted/all/b02/bundles/jtreg_bin-4.2.zip/jtreg/lib/jtreg.jar patch: java.base /opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/patches/java.base Test Layer class path: /opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi /opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi /opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun /opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun rerun: DISPLAY=scaaa563.us.oracle.com:514 \ HOME=/opt/jprt/jprtadm \ JDK8_HOME= \ LANG=C \ LC_ALL=C \ LC_CTYPE= \ LD_LIBRARY_PATH=/opt/jprt/T/P1/100525.sspitsyn/bundles/testBundle/jdk/jtreg/native \ PATH=/bin:/usr/bin \ TZ=localtime \ /opt/jprt/T/P1/100525.sspitsyn/testproduct/solaris_x64_5.11-fastdebug/bin/java \ -Dtest.class.path.prefix=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun \ -Dtest.src=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi \ -Dtest.src.path=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun \ -Dtest.classes=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi \ -Dtest.class.path=/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun \ -Dtest.vm.opts='-ea -esa -d64 -server -Xmx512m' \ -Dtest.tool.vm.opts='-J-ea -J-esa -J-d64 -J-server -J-Xmx512m' \ -Dtest.compiler.opts= \ -Dtest.java.opts='-Duser.home=/opt/jprt/T/P1/100525.sspitsyn -Djava.io.tmpdir=/opt/jprt/T/P1/100525.sspitsyn/io/solaris_x64_5.11-fastdebug-c2-jdk_svc_sanity -d64 -server' \ -Dtest.jdk=/opt/jprt/T/P1/100525.sspitsyn/testproduct/solaris_x64_5.11-fastdebug \ -Dcompile.jdk=/opt/jprt/T/P1/100525.sspitsyn/testproduct/solaris_x64_5.11-fastdebug \ -Dtest.timeout.factor=4.0 \ -Dtest.modules=jdk.jdi \ -Dtest.nativepath=/opt/jprt/T/P1/100525.sspitsyn/bundles/testBundle/jdk/jtreg/native \ -classpath /opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun/jdi:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/testoutput/jdk_svc_sanity/JTwork/classes/com/sun:/opt/jprt/T/P1/100525.sspitsyn/s/jdk/test/com/sun:/opt/jprt/jib-data/install/java/re/jtreg/4.2/promoted/all/b02/bundles/jtreg_bin-4.2.zip/jtreg/lib/javatest.jar:/opt/jprt/jib-data/install/java/re/jtreg/4.2/promoted/all/b02/bundles/jtreg_bin-4.2.zip/jtreg/lib/jtreg.jar \ InvokeTest STDOUT: vmOpts: '-ea -esa -d64 -server -Xmx512m' javaOpts: '-Duser.home=/opt/jprt/T/P1/100525.sspitsyn -Djava.io.tmpdir=/opt/jprt/T/P1/100525.sspitsyn/io/solaris_x64_5.11-fastdebug-c2-jdk_svc_sanity -d64 -server' JVM version:9-internal JDI version: 9.0 JVM description: Java Debug Interface (Reference Implementation) version 9.0 Java Debug Wire Protocol (Reference Implementation) version 9.0 JVM Debug Interface version 9.0 JVM version 9-internal (Java HotSpot(TM) 64-Bit Server VM, mixed mode, sharing) Howdy! STDERR: [2ms] run args: [InvokeTarg] [373ms] return val = "[Z@6adca536" [374ms] toString return value : "[Z@6adca536" [376ms] return val = true [376ms] invokeVoid return value matches: true [378ms] return val = true [378ms] invokeBoolean return value matches: true [379ms] Got Exception: com.sun.jdi.InvocationException: Exception occurred in target VM com.sun.jdi.InvocationException: Exception occurred in target VM at com.sun.tools.jdi.ObjectReferenceImpl.invokeMethod(jdk.jdi@9-internal/ObjectReferenceImpl.java:436) at InvokeTest.invoke(InvokeTest.java:330) at InvokeTest.invoke(InvokeTest.java:365) at InvokeTest.invoke(InvokeTest.java:372) at InvokeTest.runTests(InvokeTest.java:457) at TestScaffold.startTests(TestScaffold.java:431) at InvokeTest.main(InvokeTest.java:310) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-internal/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-internal/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-internal/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@9-internal/Method.java:531) at com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:226) at java.lang.Thread.run(java.base@9-internal/Thread.java:843) [383ms] return val = null java.lang.NullPointerException at InvokeTest.invoke(InvokeTest.java:338) at InvokeTest.invoke(InvokeTest.java:365) at InvokeTest.invoke(InvokeTest.java:372) at InvokeTest.runTests(InvokeTest.java:457) at TestScaffold.startTests(TestScaffold.java:431) at InvokeTest.main(InvokeTest.java:310) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-internal/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-internal/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-internal/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@9-internal/Method.java:531) at com.sun.javatest.regtest.agent.MainActionHelper$SameVMRunnable.run(MainActionHelper.java:226) at java.lang.Thread.run(java.base@9-internal/Thread.java:843) JavaTest Message: Test threw exception: java.lang.NullPointerException JavaTest Message: shutting down test TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.NullPointerException
08-05-2016
I see similar behavior. Below is a jstack output fragment: ----------------- 9131 ----------------- 0x00007f82c9063d84 __pthread_cond_wait + 0xc4 0x00007f82c84e83c7 Monitor::IWait(Thread*, long) + 0x127 0x00007f82c84ea3ed Monitor::wait(bool, long, bool) + 0x1fd 0x00007f82c877b114 _ZN10JavaThread17java_suspend_selfEv.part.195 + 0xb4 0x00007f82c82cd919 JvmtiRawMonitor::raw_wait(long, bool, Thread*) + 0x229 0x00007f82c82a2a82 JvmtiEnv::RawMonitorWait(JvmtiRawMonitor*, long) + 0xa2 0x00007f82c69d9b49 debugMonitorWait + 0x29 0x00007f82c69cad6e enqueueCommand + 0x11e 0x00007f82c69c6a22 reportEvents.part.3 + 0xa2 0x00007f82c69c7046 event_callback + 0x446 0x00007f82c69c9d55 cbBreakpoint + 0xc5 0x00007f82c82bc78f JvmtiExport::post_raw_breakpoint(JavaThread*, Method*, unsigned char*) + 0x1ef 0x00007f82c80bf662 InterpreterRuntime::_breakpoint(JavaThread*, Method*, unsigned char*) + 0xa2 0x00007f82a89af712 Locked ownable synchronizers: - None . . . . . . . . . . . . . . . . . . ----------------- 9169 ----------------- 0x00007f82c9063d84 __pthread_cond_wait + 0xc4 0x00007f82c84e83c7 Monitor::IWait(Thread*, long) + 0x127 0x00007f82c84ea98f Monitor::wait(bool, long, bool) + 0x79f 0x00007f82c861d380 ReferencePendingListLockerThread::receive_and_handle_messages() + 0xb0 0x00007f82c861d8a9 ???????? 0x00007f82c878c991 JavaThread::thread_main_inner() + 0x1d1 0x00007f82c878cb3a JavaThread::run() + 0x15a 0x00007f82c854a3b2 java_start(Thread*) + 0x142 Locked ownable synchronizers: - None . . . . . . . . . . . . . . . . . . ----------------- 9202 ----------------- 0x00007f82c90640fe __pthread_cond_timedwait + 0x13e 0x00007f82c85228f1 ObjectMonitor::EnterI(Thread*) + 0x581 0x00007f82c8523147 ObjectMonitor::enter(Thread*) + 0x307 0x00007f82c872fb11 ObjectSynchronizer::slow_enter(Handle, BasicLock*, Thread*) + 0xa1 0x00007f82c872fdcb ObjectSynchronizer::fast_enter(Handle, BasicLock*, bool, Thread*) + 0x8b 0x00007f82c861d9ef ReferencePendingListLocker::lock() + 0x13f 0x00007f82c87ffdef VM_GC_Operation::doit_prologue() + 0xbf 0x00007f82c8832221 VM_G1IncCollectionPause::doit_prologue() + 0x11 0x00007f82c882d49e VMThread::execute(VM_Operation*) + 0x2ee 0x00007f82c7f4a5ee G1CollectedHeap::collect(GCCause::Cause) + 0x1ce 0x00007f82c7f56c41 G1CollectedHeap::attempt_allocation_humongous(unsigned long, unsigned int*, unsigned int*) + 0x3e1 0x00007f82c7f56f1d G1CollectedHeap::mem_allocate(unsigned long, bool*) + 0x26d 0x00007f82c7abe0d7 CollectedHeap::array_allocate(KlassHandle, int, int, Thread*) + 0x2d7 0x00007f82c87b0864 TypeArrayKlass::allocate_common(int, bool, Thread*) + 0x1d4 0x00007f82c8179c3d jni_NewByteArray + 0x16d 0x00007f82c69b5d20 newInstance + 0x430 0x00007f82c69c3fd8 debugLoop_run + 0x298 0x00007f82c69d661e attachThread + 0x2e 0x00007f82c82c6093 JvmtiAgentThread::call_start_function() + 0x153 0x00007f82c878c991 JavaThread::thread_main_inner() + 0x1d1 0x00007f82c878cb3a JavaThread::run() + 0x15a 0x00007f82c854a3b2 java_start(Thread*) + 0x142 Locked ownable synchronizers: - None
05-05-2016
Note that I'm still seeing fairly rare cases (1 out of 1000) where OomDebugTest would timeout. The debugger waits for the reply from the debuggee which is in process of fulfilling a newInstance() request of a primitive array. However that reply never seems to come, because for some reason the call to jni_NewByteArray never completes? This looks like an issue in GC to me and not related to this bug. Here is the jstack snippet of the hung OomDebugTestTarg JVM process: [...] ----------------- 28829 ----------------- 0x00007efeb8e9cb10 __pthread_cond_wait + 0xc0 0x00007efeb7f3419f Monitor::IWait(Thread*, long) + 0xef 0x00007efeb7f3541e Monitor::wait(bool, long, bool) + 0x22e 0x00007efeb7fff7aa ReferencePendingListLockerThread::receive_and_handle_messages() + 0x4a 0x00007efeb7fff879 ???????? 0x00007efeb80dd227 JavaThread::thread_main_inner() + 0x1e7 0x00007efeb7f6f430 java_start(Thread*) + 0xf0 Locked ownable synchronizers: - None [...] ----------------- 28837 ----------------- 0x00007efeb8e9ceb9 __pthread_cond_timedwait + 0x129 0x00007efeb7f56bf3 ObjectMonitor::EnterI(Thread*) + 0x3c3 0x00007efeb7f572c0 ObjectMonitor::enter(Thread*) + 0x320 0x00007efeb7fff3ea ReferencePendingListLocker::lock() + 0xca 0x00007efeb813af3f VM_GC_Operation::doit_prologue() + 0x6f 0x00007efeb8149be1 VM_G1IncCollectionPause::doit_prologue() + 0x11 0x00007efeb814818e VMThread::execute(VM_Operation*) + 0x22e 0x00007efeb7bd4fb5 G1CollectedHeap::collect(GCCause::Cause) + 0x95 0x00007efeb7bdb28a G1CollectedHeap::attempt_allocation_humongous(unsigned long, unsigned int*, unsigned int*) + 0x2ea 0x00007efeb7bdb519 G1CollectedHeap::mem_allocate(unsigned long, bool*) + 0x219 0x00007efeb80f6501 TypeArrayKlass::allocate_common(int, bool, Thread*) + 0xa1 0x00007efeb7ce5713 jni_NewByteArray + 0xa3 0x00007efeb641ab56 newInstance + 0x496 0x00007efeb6428fd8 debugLoop_run + 0x268 0x00007efeb643b075 attachThread + 0x25 0x00007efeb7dbaa00 JvmtiAgentThread::start_function_wrapper(JavaThread*, Thread*) + 0xb0 0x00007efeb80dd227 JavaThread::thread_main_inner() + 0x1e7 0x00007efeb7f6f430 java_start(Thread*) + 0xf0 Locked ownable synchronizers: - None Interestingly enough it always seems to get stuck in test2() which instantiates larg-ish primitive byte arrays in the debuggee.
28-04-2016
Some clarifications. 1.) The original patch for JDK-4858370 did not hold the eventHandler lock and neither the invoker lock as the rest of invoker_completeInvokeRequest() does. This caused test failures in com/sun/jdi/InterfaceMethodsTest.java and others. Grabbing the locks again in invoker_completeInvokeRequest() before clearing global references fixes this. 2.) Even though there are cases where global references are saved while not holding the invoker lock (see invoker_doInvoke() and related impls such as invokeStatic()) this seems to be OK since invokes need to go through invoker_requestInvoke() and finish via completeInvokeRequest() which grabs the lock again. Grabbing the invoker lock too coarsely while doing the invoke may cause deadlocks. As seen during review of the first version of the fix for this bug.
28-04-2016
Posted patch for review: http://mail.openjdk.java.net/pipermail/serviceability-dev/2016-April/019410.html
15-04-2016
With the latest patch com/sun/jdi/InterfaceMethodsTest.java and com/sun/jdi/InvokeTest.java did not fail for 1300 runs.
12-04-2016
Improved fix, with locking during do_invoke()
12-04-2016
With the attached patch, com/sun/jdi/InterfaceMethodsTest.java passed ~1250 times. But there also was one failure. Not sure if it's related to the third issue described above or the second. Also, I seem to get fairly frequent failures in OomDebugTest (mostly SIGSEGV in the jvm) when JDI's newInstance() is being called. I'll see if acquiring the locks in invoke* functions fixes that.
08-04-2016
Current WIP fix.
08-04-2016
@Serguei This isn't the whole story, unfortunately. First, deleteGlobalRefs() needs to be called after the reply has been sent. Otherwise some variables needed for the reply get cleared. So my comment above is incorrect. Second, when we do clear global references after the reply has been sent this also clears global references created via JDI for example with ClassType.newInstance. This will make the code more susceptible to run into the issue as described in JDK-4257193. Finally, all invoker* incarnations, such as invokeConstructor, don't hold the invokelock either, even though, they are all saving global refs pointed to by the current request, which a fix for this issue would delete. There seems to be a possibility for races.
08-04-2016
[To Severin G.: ] I think, you spotted the issue. Feel free to re-assign the bug to yourself if you want to persist.
07-04-2016
It appears the initial fix for JDK-4858370 deleted global references while *not* holding the invoker lock, but I think it should. That is, the callsite of deleteGlobalRefs() needs to move up to to before the invoker lock is released. I'm currently having issues with the test (OomDebugTest), but I'll report back once I have that issue resolved.
07-04-2016