Bug ID: JDK-8237544 Failed to create heap dump file with "jhsdb jmap"

Type: Bug
Component: core-svc
Sub-Component: tools
Affected Version: 11.0.6

Priority: P3
Status: Closed
Resolution: Incomplete
OS: linux
CPU: x86_64

Submitted: 2020-01-20
Updated: 2020-02-27
Resolved: 2020-02-27

ADDITIONAL SYSTEM INFORMATION :
CentOS release 6.9 (Final)

openjdk 11.0.6 2020-01-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.6+10, mixed mode)


A DESCRIPTION OF THE PROBLEM :
Failed to create heap dump with "jhsdb jmap". See exact commands and errors in "Actual Result".
It failed often (but sometimes works). It failed in both cases: connect to pid and connect to core dump file.
Interesting that same command is failing with 3 different errors (probably depends on logic that executed in application at specific time):
"java.lang.IndexOutOfBoundsException: bad SID 0", "sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000100118c40", "sun.jvm.hotspot.utilities.AssertionFailure: can not get class data for java/lang/invoke/LambdaForm$MH0x00000001010bbc40" 

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
- Run java app
- Create heap dump with jhsdb (core file mode and pid mode)

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Heap dump created.
ACTUAL -
Error1:
$ /opt/app/java/bin/jhsdb jmap --binaryheap --dumpfile hd.hprof --exe /opt/app/java/bin/java --core core.16506 
Attaching to core core.16506 from executable /opt/app/java/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.6+10
Exception in thread "main" java.lang.IndexOutOfBoundsException: bad SID 0
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.vmSymbols.symbolAt(vmSymbols.java:58)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.InstanceKlass.getFieldName(InstanceKlass.java:353)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.InstanceKlass.<init>(InstanceKlass.java:153)
	at jdk.internal.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VMObjectFactory.newObject(VMObjectFactory.java:58)
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:104)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:73)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeClass(HeapHprofBinWriter.java:608)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:83)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:355)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:173)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:52)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:440)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:369)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:538)

Error2:
$ /opt/app/java/bin/jhsdb jmap --binaryheap --dumpfile hd.hprof --exe /opt/app/java/bin/java --core core.16506 
Attaching to core core.16506 from executable /opt/app/java/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.6+10
Exception in thread "main" sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000100118c40
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:109)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:73)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeClass(HeapHprofBinWriter.java:608)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:83)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:355)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:173)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:52)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:440)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:369)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:538)

Error3:
$ /opt/app/java/bin/jhsdb jmap --binaryheap --dumpfile b1.hprof --exe /opt/app/java/bin/java --core core_b1.25274 
Attaching to core core_b1.25274 from executable /opt/app/java/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.5+10
Exception in thread "main" sun.jvm.hotspot.utilities.AssertionFailure: can not get class data for java/lang/invoke/LambdaForm$MH0x00000001010bbc40
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeInstance(HeapHprofBinWriter.java:940)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:96)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:355)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:173)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:52)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:440)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:329)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:488)

Same error when connect to pid:
$ /opt/app/java/bin/jhsdb jmap --binaryheap --dumpfile hd1.hprof --pid 16506 
Attaching to process ID 16506, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.6+10
Exception in thread "main" sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000100118c40
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:109)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:73)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeClass(HeapHprofBinWriter.java:608)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:83)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:355)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:173)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:52)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:440)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:369)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:538)


---------- BEGIN SOURCE ----------
It is big application. Any idea how to troubleshot it and understand what java code lead to those errors? then I can build simple app for reproduce problem.
---------- END SOURCE ----------

FREQUENCY : often

Closing as Incomplete, as additional information to reproduce this issue is not received from filer.
27-02-2020
I'm not sure if all of these failures are related.
22-01-2020
Thanks [~cjplummer] for the clarification on ".WrongTypeException: No suitable match" exception, There is another issue here "java.lang.IndexOutOfBoundsException: bad SID 0" possibly Symbol ID is not in the range. If you think both are related or in case of no response from submitter, i will close this as duplicate.
22-01-2020
This looks like a dup of JDK-8235220. It is 100% reproducible with ClhsdbScanOops, which is currently problem listed.
21-01-2020
This looks like we encountered bad instance of class file, Please share the vm configurations for java application for which you are trying to take the heap dump. From the report it looks like it failed on 11.0.6 and 11.0.5, does this worked before? We have made couple of fixes in jdk14, could you please verify this issue against jdk14 early access build http://jdk.java.net/14/ Please provide above stated details in additional information link.
20-01-2020