JDK-8318754 : [macosx-aarch64] SA tests fail on macosx-aarch64 with various address exceptions or AIOOBE.
  • Type: Bug
  • Component: hotspot
  • Sub-Component: svc-agent
  • Affected Version: 22,23
  • Priority: P4
  • Status: Open
  • Resolution: Unresolved
  • OS: os_x
  • CPU: aarch64
  • Submitted: 2023-10-24
  • Updated: 2024-07-27
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Duplicate :  
JDK-8314453 - serviceability/sa/ClhsdbCDSCore.java fails with "'Common-Cleaner' missing from stdout/stderr"
Duplicate :  
JDK-8322148 - UnmappedAddressException walking stacks in serviceability/sa/ClhsdbPstack.java#core
Duplicate :  
JDK-8321684 - Test serviceability/sa/ClhsdbFindPC.java#no-xcomp-core failed
Relates :  
JDK-8314550 - [macosx-aarch64] serviceability/sa/TestJmapCore.java fails with "sun.jvm.hotspot.debugger.UnmappedAddressException: 801000800"
Relates :  
JDK-8284045 - serviceability/sa/ClhsdbFindPC.java#xcomp-core is failing - sun.jvm.hotspot.debugger.UnalignedAddressException: 8baadbabe
Sub Tasks
JDK-8335291 :  
Problem list all SA core file tests on macosx-aarch64 due to JDK-8318754 - Resolved
Description
The following test failed in the JDK22 CI:

serviceability/sa/TestJmapCore.java

----------System.err:(42/3125)----------
sun.jvm.hotspot.debugger.UnmappedAddressException: c801000800
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.PageCache.getLong(PageCache.java:100)
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.DebuggerBase.readCInteger(DebuggerBase.java:356)
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:382)
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.bsd.BsdDebuggerLocal.readAddress(BsdDebuggerLocal.java:421)
at jdk.hotspot.agent/sun.jvm.hotspot.debugger.bsd.BsdAddress.getAddressAt(BsdAddress.java:73)
at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:238)
at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:104)
at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:78)
at jdk.hotspot.agent/sun.jvm.hotspot.oops.MetadataField.getValue(MetadataField.java:43)
at jdk.hotspot.agent/sun.jvm.hotspot.oops.MetadataField.getValue(MetadataField.java:40)
at jdk.hotspot.agent/sun.jvm.hotspot.classfile.ClassLoaderData.getKlasses(ClassLoaderData.java:82)
at jdk.hotspot.agent/sun.jvm.hotspot.classfile.ClassLoaderData.classesDo(ClassLoaderData.java:101)
at jdk.hotspot.agent/sun.jvm.hotspot.classfile.ClassLoaderDataGraph.classesDo(ClassLoaderDataGraph.java:84)
at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeSymbols(HeapHprofBinWriter.java:1206)
at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:454)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:216)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:103)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:278)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:241)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:134)
at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:202)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:340)
at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:500)

java.io.EOFException
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:210)
at java.base/java.io.DataInputStream.readInt(DataInputStream.java:385)
at jdk.test.lib.hprof.parser.Reader.readFile(Reader.java:88)
at jdk.test.lib.hprof.HprofParser.parse(HprofParser.java:99)
at jdk.test.lib.hprof.HprofParser.parse(HprofParser.java:66)
at TestJmapCore.test(TestJmapCore.java:110)
at TestJmapCore.main(TestJmapCore.java:70)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:333)
at java.base/java.lang.Thread.run(Thread.java:1570)

JavaTest Message: Test threw exception: java.io.EOFException
JavaTest Message: shutting down test

result: Failed. Execution failed: `main' threw exception: java.io.EOFException


Please note that this is not a ZGC test run. 

[~cjplummer] said the following bug was a previous attempt to fix
this issue:

    JDK-8314550 [macosx-aarch64] serviceability/sa/TestJmapCore.java fails with "sun.jvm.hotspot.debugger.UnmappedAddressException: 801000800" 

and requested that a new bug be filed.
Comments
The SA core file tests are now problem listed on macosx-aarch64, so we shouldn't be seeing anymore failure.
27-07-2024
[~dholmes] > Lots of ArrayIndexOutOfBoundsException before the UnmappedAddressException in some cases. Unfortunately the first exception is pruned from the log due to too many log message. My guess is the first exception is UnmappedAddressException due to macosx-aarch64 issue of the core file sometimes missing valid pages.
24-06-2024
Lots of ArrayIndexOutOfBoundsException before the UnmappedAddressException in some cases.
23-06-2024
Not that I know of, but this CR and other related ones seem to come in spurts. I'm not sure why.
18-01-2024
Here's a log file snippet for the jdk-23+6-408-tier2 sighting: serviceability/sa/TestJmapCoreMetaspace.java #section:driver ----------messages:(7/258)---------- command: driver TestJmapCore run metaspace reason: User specified action: run driver/timeout=480 TestJmapCore run metaspace started: Thu Jan 18 17:14:27 GMT 2024 Mode: agentvm Agent id: 7 finished: Thu Jan 18 17:14:32 GMT 2024 elapsed time (seconds): 5.446 ----------configuration:(14/2046)---------- <snip> ----------System.out:(33/4593)---------- Command line: [/System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-23+6-408/macosx-aarch64-debug.jdk/jdk-23/fastdebug/bin/java -cp /System/Volumes/Data/mesos/work_dir/slaves/73e57426-9086-438c-bf1c-51bfaf1790ad-S29298/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/bc9b54f1-b2ec-4a86-b925-6e15c59444dd/runs/8afb80e6-c0ee-4fce-a272-fb4cd462a157/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_tier2_serviceability/classes/1/serviceability/sa/TestJmapCoreMetaspace.d:/System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-23+6-408/src.full/open/test/hotspot/jtreg/serviceability/sa:/System/Volumes/Data/mesos/work_dir/slaves/73e57426-9086-438c-bf1c-51bfaf1790ad-S29298/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/bc9b54f1-b2ec-4a86-b925-6e15c59444dd/runs/8afb80e6-c0ee-4fce-a272-fb4cd462a157/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_tier2_serviceability/classes/1/test/lib:/System/Volumes/Data/mesos/work_dir/jib-master/install/jtreg/7.3.1/1/bundles/jtreg-7.3.1+1.zip/jtreg/lib/jtreg.jar:/System/Volumes/Data/mesos/work_dir/jib-master/install/jtreg/7.3.1/1/bundles/jtreg-7.3.1+1.zip/jtreg/lib/junit-platform-console-standalone-1.9.2.jar:/System/Volumes/Data/mesos/work_dir/jib-master/install/jtreg/7.3.1/1/bundles/jtreg-7.3.1+1.zip/jtreg/lib/testng-7.3.0.jar:/System/Volumes/Data/mesos/work_dir/jib-master/install/jtreg/7.3.1/1/bundles/jtreg-7.3.1+1.zip/jtreg/lib/jcommander-1.82.jar:/System/Volumes/Data/mesos/work_dir/jib-master/install/jtreg/7.3.1/1/bundles/jtreg-7.3.1+1.zip/jtreg/lib/guice-5.1.0.jar -XX:MaxRAMPercentage=6.25 -Dtest.boot.jdk=/System/Volumes/Data/mesos/work_dir/jib-master/install/jdk/21/35/bundles/macos-aarch64/jdk-21_macos-aarch64_bin.tar.gz/jdk-21.jdk/Contents/Home -Djava.io.tmpdir=/System/Volumes/Data/mesos/work_dir/slaves/73e57426-9086-438c-bf1c-51bfaf1790ad-S29298/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/bc9b54f1-b2ec-4a86-b925-6e15c59444dd/runs/8afb80e6-c0ee-4fce-a272-fb4cd462a157/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_tier2_serviceability/tmp -XX:+CreateCoredumpOnCrash -Xmx512m -XX:MaxMetaspaceSize=64m -XX:+CrashOnOutOfMemoryError -XX:-VerifyDependencies -XX:+AlwaysPreTouch TestJmapCore metaspace ] [2024-01-18T17:14:27.575784Z] Gathering output for process 2156 [2024-01-18T17:14:27.579162Z] Waiting for completion for process 2156 [2024-01-18T17:14:27.579553Z] Waiting for completion finished for process 2156 Output and diagnostic info for process 2156 was saved into 'pid-2156-output.log' Run test with ulimit -c: unlimited [2024-01-18T17:14:27.587424Z] Gathering output for process 2157 [2024-01-18T17:14:27.588527Z] Waiting for completion for process 2157 [2024-01-18T17:14:32.153421Z] Waiting for completion finished for process 2157 Output and diagnostic info for process 2157 was saved into 'pid-2157-output.log' crashOutputString = [Aborting due to java.lang.OutOfMemoryError: Metaspace # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (/System/Volumes/Data/mesos/work_dir/slaves/b8be9c31-0e0c-4781-900b-aa9694854b6a-S5319/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/c3606279-16f7-4115-9d2d-7c1665d30127/runs/b942d212-3851-4304-9f66-b3151eca79c0/workspace/open/src/hotspot/share/utilities/debug.cpp:271), pid=2158, tid=6659 # fatal error: OutOfMemory encountered: Metaspace # # JRE version: Java(TM) SE Runtime Environment (23.0+6) (fastdebug build 23-ea+6-408) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 23-ea+6-408, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64) # Core dump will be written. Default location: core.2158 # # An error report file with more information is saved as: # /System/Volumes/Data/mesos/work_dir/slaves/73e57426-9086-438c-bf1c-51bfaf1790ad-S29298/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/bc9b54f1-b2ec-4a86-b925-6e15c59444dd/runs/8afb80e6-c0ee-4fce-a272-fb4cd462a157/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_tier2_serviceability/scratch/0/hs_err_pid2158.log [1.657s][warning][os] Loading hsdis library failed ] getCoreFileLocation found stringWithLocation = core.2158 Found core file core.2158, size = 4171mb [2024-01-18T17:14:32.175565Z] Gathering output for process 2165 Attaching to core core.2158 from executable /System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-23+6-408/macosx-aarch64-debug.jdk/jdk-23/fastdebug/bin/java, please wait... Debugger attached successfully. Server compiler detected. JVM version is 23-ea+6-408 ----------System.err:(42/3109)---------- sun.jvm.hotspot.debugger.UnmappedAddressException: 7ff00323fdd0 at jdk.hotspot.agent/sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.PageCache.getLong(PageCache.java:100) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.DebuggerBase.readCInteger(DebuggerBase.java:356) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:382) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.bsd.BsdDebuggerLocal.readAddress(BsdDebuggerLocal.java:421) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.bsd.BsdAddress.getAddressAt(BsdAddress.java:73) at jdk.hotspot.agent/sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:238) at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:104) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:78) at jdk.hotspot.agent/sun.jvm.hotspot.oops.MetadataField.getValue(MetadataField.java:43) at jdk.hotspot.agent/sun.jvm.hotspot.oops.MetadataField.getValue(MetadataField.java:40) at jdk.hotspot.agent/sun.jvm.hotspot.oops.Klass.getNextLinkKlass(Klass.java:125) at jdk.hotspot.agent/sun.jvm.hotspot.classfile.ClassLoaderData.classesDo(ClassLoaderData.java:101) at jdk.hotspot.agent/sun.jvm.hotspot.classfile.ClassLoaderDataGraph.classesDo(ClassLoaderDataGraph.java:84) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeSymbols(HeapHprofBinWriter.java:1206) at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:454) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:216) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:103) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:278) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:241) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:134) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:202) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:340) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:500) java.io.EOFException at java.base/java.io.DataInputStream.readFully(DataInputStream.java:210) at java.base/java.io.DataInputStream.readInt(DataInputStream.java:385) at jdk.test.lib.hprof.parser.Reader.readFile(Reader.java:88) at jdk.test.lib.hprof.HprofParser.parse(HprofParser.java:99) at jdk.test.lib.hprof.HprofParser.parse(HprofParser.java:66) at TestJmapCore.test(TestJmapCore.java:110) at TestJmapCore.main(TestJmapCore.java:70) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:333) at java.base/java.lang.Thread.run(Thread.java:1575) JavaTest Message: Test threw exception: java.io.EOFException JavaTest Message: shutting down test result: Failed. Execution failed: `main' threw exception: java.io.EOFException The "fatal error: OutOfMemory encountered: Metaspace" is intentional and expected. The failure mode for the test is: ----------System.err:(42/3109)---------- sun.jvm.hotspot.debugger.UnmappedAddressException: 7ff00323fdd0 at jdk.hotspot.agent/sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.PageCache.getLong(PageCache.java:100) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.DebuggerBase.readCInteger(DebuggerBase.java:356)
18-01-2024
JDK-8322148 demonstrates this same issue, but with malloc'd memory that is clearly being touch. In the case of JDK-8322148, the memory pointed to by _java_thread_list is not in the core file: Threads class SMR info: _java_thread_list=0x000060001aeeb710, length=11, elements={ (lldb) x 0x000060009aeeb710 error: core file does not contain 0x60009aeeb710 So I don't think the proposal for CDS to use malloc instead of mmap when AlwaysPreTouch is enabled is going to help.
03-01-2024
For some reason even though +AlwaysPreTouch is causing all the pages of the CDS archive to be touched (that is what JDK-8314550 accomplished), sometimes some pages still don't end up in the core file (this is once again only a macosx-aarch issue). Ioi's suggestion is to instead not rely on mmap when using +AlwaysPreTouch. static bool use_windows_memory_mapping() { const bool is_windows = (NOT_WINDOWS(false) WINDOWS_ONLY(true)); //const bool is_windows = true; // enable this to allow testing the windows mmap semantics on Linux, etc. - return is_windows; + return is_windows || AlwaysPreTouch; } This is basically the original fix Ioi had suggest for JDK-8314550, but since the code was really intended for Windows support and this fix was somewhat hackish, we went with instead actually touching each mmap'd page. If we go with the above change, the "window-isms" should be removed. The suggestion is to call it FileMapInfo::use_memory_mapping(), and although the name is similar, it will actually have the opposite semantics since "windows memory mapping" implies NOT using mmap, and the plan is for FileMapInfo::use_memory_mapping() is to return true if mmap should be used.
25-10-2023