JDK-8158237 : JVMTI hides critical debug information for memory leak tracing
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jvmti
  • Affected Version: 8,9
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: windows_2008
  • CPU: x86
  • Submitted: 2016-05-18
  • Updated: 2022-06-01
  • Resolved: 2016-06-13
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 9
9 b126Fixed
Related Reports
Relates :  
JDK-8146961 - Fix PermGen memory leaks caused by static final Exceptions
Relates :  
JDK-8033735 - make Throwable.backtrace visible to Class.getDeclaredField again
Relates :  
JDK-4446677 - VM crash during debugging.
Description
FULL PRODUCT VERSION :
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
The fix for https://bugs.openjdk.java.net/browse/JDK-4496456 completely hid the java.lang.Throwable.backtrace field from the JVMTI API. This creates problems for tools such as YourKit that use the the JVMTI since it prevents an entire class of memory leaks from being diagnosed.

https://bugs.openjdk.java.net/browse/JDK-8146961 is a memory leak (actually two memory leaks) in the XML processing caused by the use of static fields of type RuntimeException. If initialization of these fields is triggered by a class loaded by a temporary / disposable class loader (such as a web application class loader in a JavaEE environment), a reference chain is created that pins the class loader in memory. Because of the fix applied for https://bugs.openjdk.java.net/browse/JDK-4496456 it is impossible to trace the GC roots for this leak.

This reference chain passes through the Throwable.backtrace field because of the reference it holds to the class (and its class loader) that triggered the initialization. Because the backtrace field is hidden from JVMTI, YourKit (and other profilers like Eclipse MAT) is unable to identify the GC root. This makes it next to impossible to diagnose the root cause of the memory leak.

More generally, any memory leak triggered by the use of a static exception field is going to suffer from the same problem and be very difficult to diagnose.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Reproduce memory leak as per https://bugs.openjdk.java.net/browse/JDK-8146961 and then use a JVMTI based profiler to trace the GC roots for the leaking class loader.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The GC root should be traceable via Throwable.backtrace.
ACTUAL -
No GC root is visible.

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
YourKit supports an alternative, but limited in functionality, format for memory snapshots. If the HPROF format is used the GC root is visible.
Comments
URL: http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/dfcdb0a45822 User: lana Date: 2016-07-06 20:17:35 +0000
06-07-2016
URL: http://hg.openjdk.java.net/jdk9/jdk9/hotspot/rev/631fb590133e User: lana Date: 2016-07-06 20:17:31 +0000
06-07-2016
URL: http://hg.openjdk.java.net/jdk9/hs/hotspot/rev/631fb590133e User: coleenp Date: 2016-06-13 15:31:01 +0000
13-06-2016
URL: http://hg.openjdk.java.net/jdk9/hs/jdk/rev/dfcdb0a45822 User: coleenp Date: 2016-06-13 15:30:59 +0000
13-06-2016
Assigning back to you, please push the patch...
13-06-2016
Great news. Thanks.
13-06-2016
Moving from core-svc/debugger -> hotspot/jvmti
10-06-2016
Hi [~fmatte], the patch is essentially this. It would be nice if #2 they could test it out for us. I've done some investigation and there is no longer a reason that we need to filter backtrace anymore. diff --git a/src/share/vm/runtime/reflectionUtils.cpp b/src/share/vm/runtime/reflectionUtils.cpp --- a/src/share/vm/runtime/reflectionUtils.cpp +++ b/src/share/vm/runtime/reflectionUtils.cpp @@ -73,10 +73,7 @@ void FilteredFieldsMap::initialize() { - int offset; - offset = java_lang_Throwable::get_backtrace_offset(); - _filtered_fields->append(new FilteredField(SystemDictionary::Throwable_klass(), offset)); - offset = reflect_ConstantPool::oop_offset(); + int offset = reflect_ConstantPool::oop_offset(); _filtered_fields->append(new FilteredField(SystemDictionary::reflect_ConstantPool_klass(), offset)); offset = reflect_UnsafeStaticFieldAccessorImpl::base_offset(); _filtered_fields->append(new FilteredField(SystemDictionary::reflect_UnsafeStaticFieldAccessorImpl_klass(), offset));
09-06-2016
What is "YourKit"? I can't find any JVMTI code that hides backtrace. I found it in reflectionUtils.hpp void FilteredFieldsMap::initialize() { int offset; offset = java_lang_Throwable::get_backtrace_offset(); _filtered_fields->append(new FilteredField(SystemDictionary::Throwable_klass(), offset)); ... I don't know why this is filtered here though. ^AD 297 // We want to filter out java.lang.Throwable.backtrace (see 4446677). // It contains some methodOops that aren't quite real Objects. Calling certain // methods on these pseudo Objects can cause segvs. // This code assumes that offsets are never negative. // int backtraceOffset = k == SystemDictionary::throwable_klass()? java_lang_Throwable::get_backtrace_offset(): -1; ^AE 297
07-06-2016
Is the memory leak still present or the problem that you can't trace _backtrace field through the leak detector? The latter has been fixed.
03-06-2016
Submitter told it is still reproducible JDK9 build 120 and 121
03-06-2016
Reply got from submitter: Sorry to be the bearer of bad news, but this bug is NOT fixed. I've just tested this with b120 and b121 and in both cases the memory leak is traceable if I use an HPROF format heap dump but not if I use the YourKit format heap dump obtained via JVMTI.
03-06-2016
Closing duplicate of JDK-8033735
01-06-2016
I fixed this in 9. I think this report is a duplicate.
01-06-2016
This looks like issue, as unable to find the root for the leaking class loader.... this - value: org.apache.catalina.loader.WebappClassLoader #3 <- <classLoader> - class: com.example.DOMNormalizerLeakServlet, value: org.apache.catalina.loader.WebappClassLoader #3 <- [2] - class: java.lang.Object[], value: com.example.DOMNormalizerLeakServlet class DOMNormalizerLeakServlet <- [2] - class: java.lang.Object[], value: java.lang.Object[] #4319 <- backtrace - class: java.lang.RuntimeException, value: java.lang.Object[] #4318 <- abort (sticky class) - class: com.sun.org.apache.xml.internal.serialize.DOMSerializerImpl, value: java.lang.RuntimeException #1
31-05-2016