JDK-8241158 : SA TestHeapDumpForInvokeDynamic.java fails when CDS archive is relocated
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 15
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: os_x
  • CPU: x86_64
  • Submitted: 2020-03-18
  • Updated: 2020-04-27
  • Resolved: 2020-04-21
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 15
15 b20Fixed
Related Reports
Relates :  
JDK-8174994 - SA: clhsdb printmdo throws WrongTypeException when attached to a process with CDS
Relates :  
JDK-8235220 - ClhsdbScanOops.java fails with sun.jvm.hotspot.types.WrongTypeException
Relates :  
JDK-8231610 - Relocate the CDS archive if it cannot be mapped to the requested address
Relates :  
JDK-8243210 - ClhsdbScanOops fails with NullPointerException in FileMapHeader.inCopiedVtableSpace
Relates :  
JDK-8238268 - Many SA tests are not running on OSX because they do not attempt to use sudo when available
Sub Tasks
JDK-8241162 :  
ProblemList serviceability/sa/TestHeapDumpForInvokeDynamic.java on OSX - Resolved
JDK-8242153 :  
ProblemList serviceability/sa/ClhsdbDumpheap.java on OSX - Resolved
Description
The following test fails in the JDK15 CI:

serviceability/sa/TestHeapDumpForInvokeDynamic.java

Here's a snippet from the log file:

----------System.out:(34/3137)----------
[0.032s][info][cds] Archive was created with UseCompressedOops = 1, UseCompressedClassPointers = 1
[0.032s][debug][cds] Reserved archive_space_rs     [0x0000000800000000 - 0x0000000800b50000] (11862016) bytes
[0.032s][debug][cds] Reserved class_space_rs [0x0000000800b50000 - 0x0000000840b50000] (1073741824) bytes
[0.032s][info ][cds] Mapped static  region #0 at base 0x0000000800000000 top 0x0000000800007000 (MiscCode)
[0.032s][info ][cds] Mapped static  region #1 at base 0x0000000800007000 top 0x0000000800430000 (ReadWrite)
[0.032s][info ][cds] Mapped static  region #2 at base 0x0000000800430000 top 0x0000000800b50000 (ReadOnly)
[0.032s][info ][cds] ArchiveRelocationMode == 1: always map archive(s) at an alternative address
[0.032s][info ][cds] Unmapping region #0 at base 0x0000000800000000 (MiscCode)
[0.032s][info ][cds] Unmapping region #1 at base 0x0000000800007000 (ReadWrite)
[0.032s][info ][cds] Unmapping region #2 at base 0x0000000800430000 (ReadOnly)
[0.032s][debug][cds] Released shared space (archive) 0x0000000800000000
[0.032s][debug][cds] Released shared space (classes) 0x0000000800b50000
[0.033s][info ][cds] Try to map archive(s) at an alternative address
[0.033s][debug][cds] Reserved archive_space_rs     [0x0000000125d31000 - 0x0000000126881000] (11862016) bytes
[0.033s][debug][cds] Reserved class_space_rs [0x0000000126881000 - 0x0000000166881000] (1073741824) bytes
[0.033s][info ][cds] Mapped static  region #0 at base 0x0000000125d31000 top 0x0000000125d38000 (MiscCode)
[0.033s][info ][cds] Mapped static  region #1 at base 0x0000000125d38000 top 0x0000000126161000 (ReadWrite)
[0.033s][info ][cds] Mapped static  region #2 at base 0x0000000126161000 top 0x0000000126881000 (ReadOnly)
[0.054s][info ][cds] CDS archive was created with max heap size = 128M, and the following configuration:
[0.054s][info ][cds]     narrow_klass_base = 0x0000000125d31000, narrow_klass_shift = 3
[0.054s][info ][cds]     narrow_oop_mode = 1, narrow_oop_base = 0x0000000000000000, narrow_oop_shift = 3
[0.054s][info ][cds] The current max heap size = 1968M, HeapRegion::GrainBytes = 1048576
[0.054s][info ][cds]     narrow_klass_base = 0x0000000125d31000, narrow_klass_shift = 3
[0.054s][info ][cds]     narrow_oop_mode = 1, narrow_oop_base = 0x0000000000000000, narrow_oop_shift = 3
[0.054s][info ][cds] CDS heap data relocation delta = 0 bytes
[0.054s][info ][cds] Trying to map heap data: region[4] at 0x00000007bff00000, size =   495616 bytes
[0.054s][info ][cds] Trying to map heap data: region[6] at 0x00000007bfe00000, size =   323584 bytes
Adding 'sudo -E -n' to the command.
sudo -E -n /scratch/mesos/jib-master/install/jdk-15+15-610/macosx-x64-debug.jdk/jdk-15/fastdebug/bin/jhsdb jmap --binaryheap --dumpfile lambdaHeapDump.bin --pid 22398
[2020-03-18T11:28:43.466909Z] Gathering output for process 22400
[2020-03-18T11:28:56.767855Z] Waiting for completion for process 22400
[2020-03-18T11:28:56.768266Z] Waiting for completion finished for process 22400
[2020-03-18T11:28:56.768760Z] Waiting for completion for process 22400
[2020-03-18T11:28:56.769008Z] Waiting for completion finished for process 22400
----------System.err:(84/7050)----------
Command line: ['/scratch/mesos/jib-master/install/jdk-15+15-610/macosx-x64-debug.jdk/jdk-15/fastdebug/bin/java' '-XX:+UsePerfData' '-Xmx512m' '-XX:MaxRAMPercentage=12' '-XX:+UnlockDiagnosticVMOptions' '-XX:ArchiveRelocationMode=1' '-Xlog:cds=debug' '-XX:NativeMemoryTracking=detail' '-cp' '/scratch/mesos/slaves/b0d836b1-c68c-4dbd-8b78-5085890ddd4c-S1007/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6ce3a776-c1b9-4b1a-a330-a54d7d3229e3/runs/53405c3b-8592-404a-bea6-0629dbc288ef/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_cds_relocation/classes/0/serviceability/sa/TestHeapDumpForInvokeDynamic.d:/scratch/mesos/slaves/b0d836b1-c68c-4dbd-8b78-5085890ddd4c-S1007/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6ce3a776-c1b9-4b1a-a330-a54d7d3229e3/runs/53405c3b-8592-404a-bea6-0629dbc288ef/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_cds_relocation/classes/0/test/lib' 'LingeredAppWithInvokeDynamic' '930ccb30-15a8-4f7d-86a2-4e8bfa618c18.lck' ]
 stdout: [Attaching to process ID 22398, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 15-ea+15-610
dumpStack: not java Thread returning
];
 stderr: [java.lang.NullPointerException
	at jdk.hotspot.agent/sun.jvm.hotspot.memory.FileMapInfo$FileMapHeader.inCopiedVtableSpace(FileMapInfo.java:152)
	at jdk.hotspot.agent/sun.jvm.hotspot.memory.FileMapInfo.inCopiedVtableSpace(FileMapInfo.java:132)
	at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:302)
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:74)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeClass(HeapHprofBinWriter.java:613)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:82)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:258)
	at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:101)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:445)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:262)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:225)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:321)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:406)
]
 exitValue = 1

 LingeredApp stdout: [[0.032s][info][cds] Archive was created with UseCompressedOops = 1, UseCompressedClassPointers = 1
[0.033s][debug][cds] Reserved archive_space_rs     [0x0000000800000000 - 0x0000000800b50000] (11862016) bytes
[0.033s][debug][cds] Reserved class_space_rs [0x0000000800b50000 - 0x0000000840b50000] (1073741824) bytes
[0.033s][info ][cds] Mapped static  region #0 at base 0x0000000800000000 top 0x0000000800007000 (MiscCode)
[0.033s][info ][cds] Mapped static  region #1 at base 0x0000000800007000 top 0x0000000800430000 (ReadWrite)
[0.033s][info ][cds] Mapped static  region #2 at base 0x0000000800430000 top 0x0000000800b50000 (ReadOnly)
[0.033s][info ][cds] ArchiveRelocationMode == 1: always map archive(s) at an alternative address
[0.033s][info ][cds] Unmapping region #0 at base 0x0000000800000000 (MiscCode)
[0.033s][info ][cds] Unmapping region #1 at base 0x0000000800007000 (ReadWrite)
[0.033s][info ][cds] Unmapping region #2 at base 0x0000000800430000 (ReadOnly)
[0.033s][debug][cds] Released shared space (archive) 0x0000000800000000
[0.033s][debug][cds] Released shared space (classes) 0x0000000800b50000
[0.033s][info ][cds] Try to map archive(s) at an alternative address
[0.033s][debug][cds] Reserved archive_space_rs     [0x000000012330e000 - 0x0000000123e5e000] (11862016) bytes
[0.033s][debug][cds] Reserved class_space_rs [0x0000000123e5e000 - 0x0000000163e5e000] (1073741824) bytes
[0.033s][info ][cds] Mapped static  region #0 at base 0x000000012330e000 top 0x0000000123315000 (MiscCode)
[0.033s][info ][cds] Mapped static  region #1 at base 0x0000000123315000 top 0x000000012373e000 (ReadWrite)
[0.033s][info ][cds] Mapped static  region #2 at base 0x000000012373e000 top 0x0000000123e5e000 (ReadOnly)
[0.054s][info ][cds] CDS archive was created with max heap size = 128M, and the following configuration:
[0.054s][info ][cds]     narrow_klass_base = 0x000000012330e000, narrow_klass_shift = 3
[0.054s][info ][cds]     narrow_oop_mode = 1, narrow_oop_base = 0x0000000000000000, narrow_oop_shift = 3
[0.054s][info ][cds] The current max heap size = 512M, HeapRegion::GrainBytes = 1048576
[0.054s][info ][cds]     narrow_klass_base = 0x000000012330e000, narrow_klass_shift = 3
[0.054s][info ][cds]     narrow_oop_mode = 1, narrow_oop_base = 0x0000000000000000, narrow_oop_shift = 3
[0.054s][info ][cds] CDS heap data need to be relocated because
[0.054s][info ][cds] the desired range 0x00000007bfe00000 - 0x00000007bff79000
[0.054s][info ][cds] is outside of the heap 0x00000007e0000000 - 0x0000000800000000
[0.054s][info ][cds] CDS heap data relocation delta = 1073741824 bytes
[0.054s][info ][cds] Trying to map heap data: region[4] at 0x00000007fff00000, size =   495616 bytes
[0.054s][info ][cds] Trying to map heap data: region[6] at 0x00000007ffe00000, size =   323584 bytes
Hello
Hello Hello
false
];
 LingeredApp stderr: []
 LingeredApp exitValue = 0
java.lang.RuntimeException: Expected to get exit value of [0]

	at jdk.test.lib.process.OutputAnalyzer.shouldHaveExitValue(OutputAnalyzer.java:455)
	at TestHeapDumpForInvokeDynamic.attachDumpAndVerify(TestHeapDumpForInvokeDynamic.java:106)
	at TestHeapDumpForInvokeDynamic.main(TestHeapDumpForInvokeDynamic.java:130)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
	at java.base/java.lang.Thread.run(Thread.java:832)

JavaTest Message: Test threw exception: java.lang.RuntimeException: Expected to get exit value of [0]

JavaTest Message: shutting down test

STATUS:Failed.`main' threw exception: java.lang.RuntimeException: Expected to get exit value of [0]
----------rerun:(42/6562)*----------

This failure first popped up in jdk-15+15-610-tier4 which
includes the fix for this bug: JDK-8238268
Comments
URL: https://hg.openjdk.java.net/jdk/jdk/rev/d83d3b3d8855 User: iklam Date: 2020-04-21 02:23:47 +0000
21-04-2020
ILW = MMM = P3
14-04-2020
The test passes on Linux just because we got lucky -- probably due to memory layout of the OS, SA hits a different type of exception, handles it, and limps along without giving an exception. However, the resulting heap dump is wrong: # run with -XX:ArchiveRelocationMode=1 $ grep writeClass: TestHeapDumpForInvokeDynamic.jtr.bad | wc 69 138 3519 # run without -XX:ArchiveRelocationMode=1 $ grep writeClass: TestHeapDumpForInvokeDynamic.jtr.good | wc 1329 2658 67779
11-04-2020
This is a bug in the CDS relocation code. When -XX:ArchiveRelocationMode=1 is specified, the CDS archive is mapped to an address picked by the OS (instead of the default address 0x800000000). See JDK-8231610. As a result, we need to patch the native pointers stored inside java.lang.Class instances: http://hg.openjdk.java.net/jdk/jdk/file/5ac19bd3a1e2/src/hotspot/share/classfile/javaClasses.cpp#l1257 On 64-bit VMs, this native pointer (an InstanceKlass*) is stored as a jlong on offset 16 of java.lang.Class instances. However, this patching is done only as needed -- when an archive class is loaded. I tested with revision 5ac19bd3a1e2 (with the attached diagnose.patch.txt), and the SA code fails because it sees an unpatched pointer: writeClass: sun.jvm.hotspot.oops.Instance@ffe003a8 instantiateWrapperFor:0x000000011fe92988 @ 0: 0x 45ed376301 @ 8: 0x 6ed31 @ 16: 0x 800007000 <<<< InstanceKlass* .... instantiateWrapperFor:0x0000000800007000 class sun.jvm.hotspot.debugger.bsd.BsdAddress/800007000/0x0000000800007000 sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000800007000 at jdk.hotspot.agent/sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:109) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:75) at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeClass(HeapHprofBinWriter.java:615) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:82) at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:258) at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:101) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:445) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:262) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:225) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:331) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:483) This class is never loaded during the execution of the test case, and hence the pointer is not patched. (gdb) p {char[40]}((InstanceKlass*)0x0000000800007000)->_name->_body $6 = "[Ljdk/internal/math/FDBigInteger;....." BTW, I don't know why this bug shows up only on macos and not on other platforms.
10-04-2020
The failing java.lang.Class object is inside the CDS archived heap. It's actually not reachable (because its InstanceKlass is not loaded), so strictly speaking, SA should not try to process the contents of this object. However, calculating reachability is hard, so we should make the archived heap content more friendly for walking. Instead of patching the native pointers on-demand, we can just patch all native pointers of all archived java.lang.Class objects during VM bootstrap.
10-04-2020
This occurred in the new test I just wrote to test out the new "clhsdb dumpheap" command. Test: serviceability/sa/ClhsdbDumpheap.java java.lang.NullPointerException at jdk.hotspot.agent/sun.jvm.hotspot.memory.FileMapInfo$FileMapHeader.inCopiedVtableSpace(FileMapInfo.java:152) at jdk.hotspot.agent/sun.jvm.hotspot.memory.FileMapInfo.inCopiedVtableSpace(FileMapInfo.java:132) at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:302) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:74) at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeClass(HeapHprofBinWriter.java:613) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:82) at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:258) at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:101) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:445) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182) at jdk.hotspot.agent/sun.jvm.hotspot.CommandProcessor$50.doit(CommandProcessor.java:1719) at jdk.hotspot.agent/sun.jvm.hotspot.CommandProcessor.executeCommand(CommandProcessor.java:2010) at jdk.hotspot.agent/sun.jvm.hotspot.CommandProcessor.executeCommand(CommandProcessor.java:1980) at jdk.hotspot.agent/sun.jvm.hotspot.CommandProcessor.run(CommandProcessor.java:1860) at jdk.hotspot.agent/sun.jvm.hotspot.CLHSDB.run(CLHSDB.java:99) at jdk.hotspot.agent/sun.jvm.hotspot.CLHSDB.main(CLHSDB.java:40) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runCLHSDB(SALauncher.java:280) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:483)
04-04-2020
There are at least two bugs here: [1] There's a bug in http://hg.openjdk.java.net/jdk/jdk/file/1b6cb377d024/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/memory/FileMapInfo.java#l151 public boolean inCopiedVtableSpace(Address vptrAddress) { if (vptrAddress.greaterThan(mcRegionBaseAddress) && vptrAddress.lessThanOrEqual(mcRegionEndAddress)) { return true; } return false; } But when SA reads a NULL pointer from memory, instead of returning an Address object that contains a NULL address, we get a null Address reference. E.g., http://hg.openjdk.java.net/jdk/jdk/file/1b6cb377d024/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/debugger/proc/ProcDebuggerLocal.java#l354 public ProcAddress readAddress(long address) throws UnmappedAddressException, UnalignedAddressException { long value = readAddressValue(address); return (value == 0 ? null : new ProcAddress(this, value)); } So inCopiedVtableSpace should check for null and return false. [2] Why are we getting 0x800003000 here?? stderr: [sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000800003000 at jdk.hotspot.agent/sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:74) at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67)
26-03-2020
This is similar to JDK-8235220: Error: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000800000028 sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000800000028 at jdk.hotspot.agent/sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:109) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:74) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.RobustOopDeterminator.oopLooksValid(RobustOopDeterminator.java:73) Apparently in both cases, SA finds an InstanceKlass* p, and p's vtable is at either 0x0000000800000028 or 0x0000000800003000. These look like vtables in the CDS archive, but they are not valid -- CDS vtables must be at locations found by here http://hg.openjdk.java.net/jdk/jdk/file/1b6cb377d024/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/memory/FileMapInfo.java#l160
25-03-2020
[~cjplummer] This requires more investigation. I think it's OK to problemlist for macos for now. The problem is with the address 0x0000000800003000. This is normally the an address inside the mapped CDS archive, but in this test, we simulate a mapping failure at the default address range (0x0000000800000000 ~ ...) and the archive is now mapped at (0x000000012330e000 ~ ...). So I don't know why SA is querying an address at 0x0000000800003000 0.033s][info ][cds] Mapped static region #0 at base 0x0000000800000000 top 0x0000000800007000 (MiscCode) [0.033s][info ][cds] Mapped static region #1 at base 0x0000000800007000 top 0x0000000800430000 (ReadWrite) [0.033s][info ][cds] Mapped static region #2 at base 0x0000000800430000 top 0x0000000800b50000 (ReadOnly) [0.033s][info ][cds] ArchiveRelocationMode == 1: always map archive(s) at an alternative address <<<<<<<<<<<<<<<< [0.033s][info ][cds] Unmapping region #0 at base 0x0000000800000000 (MiscCode) [0.033s][info ][cds] Unmapping region #1 at base 0x0000000800007000 (ReadWrite) [0.033s][info ][cds] Unmapping region #2 at base 0x0000000800430000 (ReadOnly) [0.033s][debug][cds] Released shared space (archive) 0x0000000800000000 [0.033s][debug][cds] Released shared space (classes) 0x0000000800b50000 [0.033s][info ][cds] Try to map archive(s) at an alternative address [0.033s][debug][cds] Reserved archive_space_rs [0x000000012330e000 - 0x0000000123e5e000] (11862016) bytes [0.033s][debug][cds] Reserved class_space_rs [0x0000000123e5e000 - 0x0000000163e5e000] (1073741824) bytes [0.033s][info ][cds] Mapped static region #0 at base 0x000000012330e000 top 0x0000000123315000 (MiscCode) <<<<<<<<<< [0.033s][info ][cds] Mapped static region #1 at base 0x0000000123315000 top 0x000000012373e000 (ReadWrite) [0.033s][info ][cds] Mapped static region #2 at base 0x000000012373e000 top 0x0000000123e5e000 (ReadOnly)
18-03-2020
[~iklam] If you have a suggestion as to what the core issue might be and how to fix, I'll take a stab at it. Otherwise I'll just problemlist it as planned.
18-03-2020
The top of the exception stack trace is: stderr: [java.lang.NullPointerException at jdk.hotspot.agent/sun.jvm.hotspot.memory.FileMapInfo$FileMapHeader.inCopiedVtableSpace(FileMapInfo.java:152) at jdk.hotspot.agent/sun.jvm.hotspot.memory.FileMapInfo.inCopiedVtableSpace(FileMapInfo.java:132) at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:302) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:74) at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67) findDynamicTypeForAddress() contains the following: if (VM.getVM().isSharingEnabled()) { // Check if the value falls in the _md_region FileMapInfo cdsFileMapInfo = VM.getVM().getFileMapInfo(); if (cdsFileMapInfo.inCopiedVtableSpace(loc1)) { <--- Line 302 return cdsFileMapInfo.getTypeForVptrAddress(loc1); } } This entire section of code was added by: changeset: 49896:ec2dd30adbc1 user: jgeorge date: Thu Apr 26 12:25:36 2018 +0530 summary: 8174994: SA: clhsdb printmdo throws WrongTypeException when attached to a process with CDS Just before this code you have: Address loc1 = addr.getAddressAt(0); The issue is that loc1 == null, so eventually this results in an NPE when passed to inCopiedVtableSpace(). I'm not certain, but I think loc1 is suppose to be the C++ vtable. If I add a check for loc1 == null and skip the inCopiedVtableSpace() call in that case, the failure then becomes: stderr: [sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000800003000 at jdk.hotspot.agent/sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:74) at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67) This exception seems to be what JDK-8174994 was all about. It seems the issue is that with the address being passed into instantiateWrapperFor() is not valid.
18-03-2020
The NPE stack: Looks like a NULL vptrAddress was passed to FileMapInfo$FileMapHeader.inCopiedVtableSpace(). stderr: [java.lang.NullPointerException at jdk.hotspot.agent/sun.jvm.hotspot.memory.FileMapInfo$FileMapHeader.inCopiedVtableSpace(FileMapInfo.java:152) at jdk.hotspot.agent/sun.jvm.hotspot.memory.FileMapInfo.inCopiedVtableSpace(FileMapInfo.java:132) at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:302) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:74) at jdk.hotspot.agent/sun.jvm.hotspot.oops.java_lang_Class.asKlass(java_lang_Class.java:67) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeClass(HeapHprofBinWriter.java:613) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter$1.doObj(AbstractHeapGraphWriter.java:82) at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:258) at jdk.hotspot.agent/sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:101) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.AbstractHeapGraphWriter.write(AbstractHeapGraphWriter.java:51) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:445) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:262) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:225) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:321) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:406) ] http://hg.openjdk.java.net/valhalla/valhalla/file/7da4d5f38f96/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/memory/FileMapInfo.java#l152 public boolean inCopiedVtableSpace(Address vptrAddress) { if (vptrAddress.greaterThan(mcRegionBaseAddress) && vptrAddress.lessThanOrEqual(mcRegionEndAddress)) { return true; } return false; }
18-03-2020
The following will produce the failure locally: make run-test CONF_NAME=macosx-x64 TEST="serviceability/sa/TestHeapDumpForInvokeDynamic.java" TEST_VM_OPTS="-XX:+UnlockDiagnosticVMOptions -XX:ArchiveRelocationMode=1"
18-03-2020
[~cjplummer] The above message is normal. If affects only the agentvm threads but do not impact the tests themselves.
18-03-2020
I just noticed the following in the jtreg harness log (not the .jtr file) [2020-03-18T11:02:32,664Z] [2020-03-18 11:02:32,664] Agent[3]: stdout: [0.004s][info][cds] Unable to use shared archive: CDS is disabled when java.base module is patched. I first noticed it in the test output when I ran from the command line with the above options. The test did fail in the same manner with these options, but the above message caught my attention seemed interesting. Not sure if it is related to the problem or not.
18-03-2020
3 times now. This test was just enabled on OSX after fixing JDK-8238268. The failure seems to be CDS related as it only happens with a "cds" task that runs that use the following options: -XX:+UnlockDiagnosticVMOptions -XX:ArchiveRelocationMode=1 -Xlog:cds=debug -XX:NativeMemoryTracking=detail I still need to confirm that the issue is contained within these options, and there isn't some other mach5 magic going on that adds more options. I will quarantine this test on OSX for now.
18-03-2020
This test has failed once (on OSX) in two Tier4 job sets in a row.
18-03-2020