JDK-8357448 : AOT crashes on linux musl with AddReads.java
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 25
  • Priority: P3
  • Status: Open
  • Resolution: Unresolved
  • OS: linux_alpine
  • CPU: x86
  • Submitted: 2025-05-21
  • Updated: 2025-05-26
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 25
25Unresolved
Related Reports
Causes :  
JDK-8354083 - Support --add-reads with -XX:+AOTClassLinking
Description
There were crashes while running runtime/cds/appcds/aotClassLinking/AddReads.java on Alpine linux. See attached hs_err file. Note that Compact Object Headers were enabled by patch for testing:
- product(bool, UseCompactObjectHeaders, false, EXPERIMENTAL,
+ product(bool, UseCompactObjectHeaders, true,

java.lang.RuntimeException: Hotspot crashed
	at jdk.test.lib.cds.CDSTestUtils.executeAndLog(CDSTestUtils.java:703)
	at jdk.test.lib.cds.CDSAppTester.executeAndCheck(CDSAppTester.java:194)
	at jdk.test.lib.cds.CDSAppTester.recordAOTConfiguration(CDSAppTester.java:250)
	at jdk.test.lib.cds.CDSAppTester.runAOTWorkflow(CDSAppTester.java:437)
	at jdk.test.lib.cds.SimpleCDSAppTester.runAOTWorkflow(SimpleCDSAppTester.java:189)
	at AddReads.test(AddReads.java:120)
	at AddReads.main(AddReads.java:190)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:335)
	at java.base/java.lang.Thread.run(Thread.java:1447)

We have also seen aotClassLinking crashes on Alpine linux without Compact Object Headers. New attachment AddReads.jtr was created without it.
The crashes seem to happen always since 2025-05-16 (after JDK-8354887 and JDK-8354083).
Comments
For completeness, ASAN on "normal" Linuxx86_64 (glibc no MUSL) generates a nice report on the issue when running the AddReads test : ==78422==ERROR: AddressSanitizer: heap-use-after-free on address 0x50d0000276c0 at pc 0x14f4c1058f7c bp 0x14f4b97d8970 sp 0x14f4b97d8130 READ of size 3 at 0x50d0000276c0 thread T1 #0 0x14f4c1058f7b (/usr/lib64/libasan.so.8+0x9ef7b) (BuildId: 4ee117fa2a132af1da9f17a0a5fe1f888398d50f) #1 0x14f4c1084e86 in vsnprintf (/usr/lib64/libasan.so.8+0xcae86) (BuildId: 4ee117fa2a132af1da9f17a0a5fe1f888398d50f) #2 0x14f4bc8f08d1 in permit_forbidden_function::vsnprintf(char*, unsigned long, char const*, __va_list_tag*) src/hotspot/share/utilities/permitForbiddenFunctions.hpp:60 #3 0x14f4bc8f08d1 in os::vsnprintf(char*, unsigned long, char const*, __va_list_tag*) src/hotspot/share/runtime/os.cpp:122 #4 0x14f4bc38aace in LogTagSet::vwrite(LogLevel::type, char const*, __va_list_tag*) src/hotspot/share/logging/logTagSet.cpp:131 #5 0x14f4ba2f8b8d in LogImpl<(LogTag::type)17, (LogTag::type)113, (LogTag::type)0, (LogTag::type)0, (LogTag::type)0, (LogTag::type)0>::vwrite(LogLevel::type, char const*, __va_list_tag*) src/hotspot/share/logging/log.hpp:148 #6 0x14f4ba2f8b8d in void LogImpl<(LogTag::type)17, (LogTag::type)113, (LogTag::type)0, (LogTag::type)0, (LogTag::type)0, (LogTag::type)0>::write<(LogLevel::type)3>(char const*, ...) src/hotspot/share/logging/log.hpp:142 #7 0x14f4ba2f81e5 in AOTClassLocationConfig::dumptime_init_helper(JavaThread*) src/hotspot/share/cds/aotClassLocation.cpp:494 #8 0x14f4ba2f89ee in AOTClassLocationConfig::dumptime_init(JavaThread*) src/hotspot/share/cds/aotClassLocation.cpp:450 #9 0x14f4bd346851 in SystemDictionary::initialize(JavaThread*) src/hotspot/share/classfile/systemDictionary.cpp:1565 #10 0x14f4bd4e0d69 in Universe::genesis(JavaThread*) src/hotspot/share/memory/universe.cpp:439 #11 0x14f4bd4e71ce in universe2_init() src/hotspot/share/memory/universe.cpp:1076 #12 0x14f4bb8830d8 in init_globals2() src/hotspot/share/runtime/init.cpp:166 #13 0x14f4bd481fb1 in Threads::create_vm(JavaVMInitArgs*, bool*) src/hotspot/share/runtime/threads.cpp:615 #14 0x14f4bbc0dc88 in JNI_CreateJavaVM_inner src/hotspot/share/prims/jni.cpp:3587 #15 0x14f4bbc0dc88 in JNI_CreateJavaVM src/hotspot/share/prims/jni.cpp:3678 #16 0x14f4c0f99633 in InitializeJVM src/java.base/share/native/libjli/java.c:1506 #17 0x14f4c0f99633 in JavaMain src/java.base/share/native/libjli/java.c:494 #18 0x14f4c0fa1e58 in ThreadJavaMain src/java.base/unix/native/libjli/java_md.c:646 #19 0x14f4c1018ff5 (/usr/lib64/libasan.so.8+0x5eff5) (BuildId: 4ee117fa2a132af1da9f17a0a5fe1f888398d50f) #20 0x14f4c0d6b6e9 in start_thread (/lib64/libpthread.so.0+0xa6e9) (BuildId: 938e42b7e407d175ee3ef9a89c038168101d330c) #21 0x14f4c0eae58e in clone (/lib64/libc.so.6+0x11858e) (BuildId: 74f77bf013a66413c77197c121955e029c32d259) 0x50d0000276c0 is located 0 bytes inside of 136-byte region [0x50d0000276c0,0x50d000027748) freed by thread T1 here: #0 0x14f4c10aff58 (/usr/lib64/libasan.so.8+0xf5f58) (BuildId: 4ee117fa2a132af1da9f17a0a5fe1f888398d50f) #1 0x14f4ba2f8189 in AOTClassLocationConfig::dumptime_init_helper(JavaThread*) src/hotspot/share/cds/aotClassLocation.cpp:493 #2 0x14f4ba2f89ee in AOTClassLocationConfig::dumptime_init(JavaThread*) src/hotspot/share/cds/aotClassLocation.cpp:450 #3 0x14f4bd346851 in SystemDictionary::initialize(JavaThread*) src/hotspot/share/classfile/systemDictionary.cpp:1565 #4 0x14f4bd4e0d69 in Universe::genesis(JavaThread*) src/hotspot/share/memory/universe.cpp:439 #5 0x14f4bd4e71ce in universe2_init() src/hotspot/share/memory/universe.cpp:1076 #6 0x14f4bb8830d8 in init_globals2() src/hotspot/share/runtime/init.cpp:166 #7 0x14f4bd481fb1 in Threads::create_vm(JavaVMInitArgs*, bool*) src/hotspot/share/runtime/threads.cpp:615 #8 0x14f4bbc0dc88 in JNI_CreateJavaVM_inner src/hotspot/share/prims/jni.cpp:3587 #9 0x14f4bbc0dc88 in JNI_CreateJavaVM src/hotspot/share/prims/jni.cpp:3678 #10 0x14f4c0f99633 in InitializeJVM src/java.base/share/native/libjli/java.c:1506 #11 0x14f4c0f99633 in JavaMain src/java.base/share/native/libjli/java.c:494 #12 0x14f4c0fa1e58 in ThreadJavaMain src/java.base/unix/native/libjli/java_md.c:646 #13 0x14f4c1018ff5 (/usr/lib64/libasan.so.8+0x5eff5) (BuildId: 4ee117fa2a132af1da9f17a0a5fe1f888398d50f) previously allocated by thread T1 here: #0 0x14f4c10b12b7 in malloc (/usr/lib64/libasan.so.8+0xf72b7) (BuildId: 4ee117fa2a132af1da9f17a0a5fe1f888398d50f) #1 0x14f4bc8f3a07 in permit_forbidden_function::malloc(unsigned long) src/hotspot/share/utilities/permitForbiddenFunctions.hpp:63 #2 0x14f4bc8f3a07 in os::malloc(unsigned long, MemTag, NativeCallStack const&) src/hotspot/share/runtime/os.cpp:659 #3 0x14f4ba2dbf2b in AllocateHeap(unsigned long, MemTag, NativeCallStack const&, AllocFailStrategy::AllocFailEnum) src/hotspot/share/memory/allocation.cpp:40 #4 0x14f4ba2dbf2b in AllocateHeap(unsigned long, MemTag, AllocFailStrategy::AllocFailEnum) src/hotspot/share/memory/allocation.cpp:50 #5 0x14f4ba2e99e6 in AOTClassLocationConfig::find_lcp(ClassLocationStream&, unsigned long&) src/hotspot/share/cds/aotClassLocation.cpp:556 #6 0x14f4ba2f7c50 in AOTClassLocationConfig::dumptime_init_helper(JavaThread*) src/hotspot/share/cds/aotClassLocation.cpp:491 #7 0x14f4ba2f89ee in AOTClassLocationConfig::dumptime_init(JavaThread*) src/hotspot/share/cds/aotClassLocation.cpp:450 #8 0x14f4bd346851 in SystemDictionary::initialize(JavaThread*) src/hotspot/share/classfile/systemDictionary.cpp:1565 #9 0x14f4bd4e0d69 in Universe::genesis(JavaThread*) src/hotspot/share/memory/universe.cpp:439 #10 0x14f4bd4e71ce in universe2_init() src/hotspot/share/memory/universe.cpp:1076 #11 0x14f4bb8830d8 in init_globals2() src/hotspot/share/runtime/init.cpp:166 #12 0x14f4bd481fb1 in Threads::create_vm(JavaVMInitArgs*, bool*) src/hotspot/share/runtime/threads.cpp:615 #13 0x14f4bbc0dc88 in JNI_CreateJavaVM_inner src/hotspot/share/prims/jni.cpp:3587 #14 0x14f4bbc0dc88 in JNI_CreateJavaVM src/hotspot/share/prims/jni.cpp:3678 #15 0x14f4c0f99633 in InitializeJVM src/java.base/share/native/libjli/java.c:1506 #16 0x14f4c0f99633 in JavaMain src/java.base/share/native/libjli/java.c:494 #17 0x14f4c0fa1e58 in ThreadJavaMain src/java.base/unix/native/libjli/java_md.c:646 #18 0x14f4c1018ff5 (/usr/lib64/libasan.so.8+0x5eff5) (BuildId: 4ee117fa2a132af1da9f17a0a5fe1f888398d50f) Thread T1 created by T0 here: #0 0x14f4c10a9191 in pthread_create (/usr/lib64/libasan.so.8+0xef191) (BuildId: 4ee117fa2a132af1da9f17a0a5fe1f888398d50f) #1 0x14f4c0fa37a8 in CallJavaMainInNewThread src/java.base/unix/native/libjli/java_md.c:687 #2 0x14f4c0f9f400 in ContinueInNewThread src/java.base/share/native/libjli/java.c:2340 #3 0x14f4c0fa0d5d in JLI_Launch src/java.base/share/native/libjli/java.c:330 #4 0x5616c21480fc in main src/java.base/share/native/launcher/main.c:150 #5 0x14f4c0dcb24c in __libc_start_main (/lib64/libc.so.6+0x3524c) (BuildId: 74f77bf013a66413c77197c121955e029c32d259) SUMMARY: AddressSanitizer: heap-use-after-free (/usr/lib64/libasan.so.8+0x9ef7b) (BuildId: 4ee117fa2a132af1da9f17a0a5fe1f888398d50f) Shadow bytes around the buggy address: 0x50d000027400: 00 00 fa fa fa fa fa fa fa fa 00 00 00 00 00 00 0x50d000027480: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa 0x50d000027500: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00 0x50d000027580: 00 00 00 00 00 fa fa fa fa fa fa fa fa fa 00 00 0x50d000027600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x50d000027680: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd 0x50d000027700: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa 0x50d000027780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x50d000027800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x50d000027880: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x50d000027900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==78422==ABORTING
26-05-2025
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk/pull/25446 Date: 2025-05-26 10:12:24 +0000
26-05-2025
This looks wrong : https://github.com/openjdk/jdk/blob/ed4cd2acd2d8bb92c296c5a860c76cffaff53add/src/hotspot/share/cds/aotClassLocation.cpp#L493 and usage the line below ; any idea why we see it now, the coding is in for quite a while as Arno stated? Maybe some other code path is executed now ?
26-05-2025
I was unable to reproduce it standalone, but got my hands on a core from our nighty runs. The issue is a use-after-free in aotClassLocation.cpp. The code was added 3 month ago with "8280682: Refactor AOT code source validation checks" - but it seems that only the new test triggered it. I will provide a pr to fix this.
26-05-2025
Thank you for checking. Yes, it has all latest AOT code changes. Is it possible to run your testing with VM flags -XX:+UnlockDiagnosticVMOptions -XX:-AOTAdapterCaching ? To switch off AOT code generation (runtime stubs are switched already). Or with next patch if you build JDK by yourself: diff --git a/src/hotspot/share/code/aotCodeCache.cpp b/src/hotspot/share/code/aotCodeCache.cpp index 4db16fdbc70..191f489d961 100644 --- a/src/hotspot/share/code/aotCodeCache.cpp +++ b/src/hotspot/share/code/aotCodeCache.cpp @@ -141,6 +141,7 @@ void AOTCodeCache::initialize() { // Disable stubs caching until JDK-8357398 is fixed. FLAG_SET_ERGO(AOTStubCaching, false); + FLAG_SET_ERGO(AOTAdapterCaching, false); if (VerifyOops) { // Disable AOT stubs caching when VerifyOops flag is on.
23-05-2025
> Can you try the latest JDK? I hade several fixes there for AOT code generation. Our last build/test cycle still showed the crashes, they are based on tha last change taken from github 8139228: JFileChooser renders file names as HTML document https://github.wdf.sap.corp/SapMachine/test-jdk/commit/917c1546f353c2814de8465d1dfad66b01561f12 (so I think Vladimirs changes are already in, this was rather recent)
23-05-2025
I have also setup a local VM running alpine linux on x86-64 but I haven't been able to recreate this issue at all. I tried with and without enabling UseCompactObjectHeaders. All tests under runtime/cds/appcds/aotClassLinking passed. I have run this multiple times.
23-05-2025
[~mdoerr] Can you try the latest JDK? I hade several fixes there for AOT code generation.
23-05-2025
Without logs from failed runs it is impossible to find the cause.
23-05-2025
May be we should enable COH by default in sources and run through mach5 testing to see if we also hit some failures.
23-05-2025
AOT code caching works only with AOT Class linking as per agreement. `add-reads-..` file is from AOTMode=record too: [3.797s][info ][aot ] Writing binary AOTConfiguration file: add-reads-with-classpath.aotconfig AOT code is not generated as expected: [3.875s][debug][aot ] ac space: 0 [ 0.0% of total] out of 0 bytes [ 0.0% used] at 0x0000000000000000
23-05-2025
This test first runs the classical static CDS archive workflow, and then the AOT workflow https://github.com/openjdk/jdk/blob/90e076b2a1ee5f91317157911e2c62a37978e93e/test/hotspot/jtreg/runtime/cds/appcds/aotClassLinking/AddReads.java#L119-L120 static SimpleCDSAppTester test(String comment, SimpleCDSAppTester tester) throws Exception { printComment(comment); return tester .setAssemblyChecker((OutputAnalyzer out) -> { out.shouldContain("Full module graph = enabled"); }) .setProductionChecker((OutputAnalyzer out) -> { out.shouldContain("use_full_module_graph = true; java.base"); }) .runStaticWorkflow() .runAOTWorkflow(); } From the attached AddReads.jtr file, the static workflow finished successfully, but we get the crash in the AOT workflow. One difference is that adapters are cached only in the AOT workflow: bool CDSConfig::is_dumping_adapters() { return (AOTAdapterCaching && is_dumping_final_static_archive()); } [~asmehra] [~kvn] Maybe this means that the problem is in adapters?
23-05-2025
Thanks! I have changed "caused by" to JDK-8354083. Both changes came in at the same day.
23-05-2025
[~chagedorn] "use -XX:-UseCompactObjectHeaders = HLM = P3" that does not make the issue go away, we have crashes with and without UseCompactObjectHeaders set.
23-05-2025
>Is this the only test that fails? Today it is the only remaining crash. >Is this crash intermittent? We saw the crash happen at least the last 3 days, every day. So rather regularly. >Can you provide instructions for reproducing the test environment so I can investigate? We have to look into this a bit more; building on Alpine myself outside our central tests/makes and running the runtime/cds/appcds/aotClassLinking/AddReads.java test did not show the crash; even running the whole hs :tier2 on my own builds did not show it. So there must be something more that is needed to show the issue.
23-05-2025
Thanks Matthias, I removed my initial ILW and leave the triaging to the runtime team.
23-05-2025
[~mdoerr] this test case (AddReads.java) was first introduced in JDK-8354083, which is integrated after JDK-8354887, so the latter might not be the cause of this bug (although JDK-8354887 causes JDK-8357398, which also shows signs of memory stomping).
22-05-2025
If you look at AddReads.jtr, it list all the log files that have been saved. Could you attach all of them (preferably in a ZIP file so they don't get mixed up_? You can search for "file=" and "[logging" in the jtr file for the names of these log files. -Xlog:class+load=debug,aot=debug,cds=debug,cds+class=debug:file=add-reads-ALL-UNNAMED.aotconfig.log: -Xlog:aot=debug,cds=debug,cds+class=debug,aot+heap=warning,cds+resolve=debug:file=add-reads-ALL-UNNAMED.aot.log::filesize=0 [logging stdout to /output_openjdk25_stage_dbgU_linuxmuslx86_64/jtreg_hotspot_tier2_work/JTwork/scratch/7/runtime.cds.appcds.aotClassLinking.AddReads.java-0012-TRAINING.stdout] [logging stderr to /output_openjdk25_stage_dbgU_linuxmuslx86_64/jtreg_hotspot_tier2_work/JTwork/scratch/7/runtime.cds.appcds.aotClassLinking.AddReads.java-0012-TRAINING.stderr] [STDERR] [logging stdout to /output_openjdk25_stage_dbgU_linuxmuslx86_64/jtreg_hotspot_tier2_work/JTwork/scratch/7/runtime.cds.appcds.aotClassLinking.AddReads.java-0013-ASSEMBLY.stdout] [logging stderr to /output_openjdk25_stage_dbgU_linuxmuslx86_64/jtreg_hotspot_tier2_work/JTwork/scratch/7/runtime.cds.appcds.aotClassLinking.AddReads.java-0013-ASSEMBLY.stderr]
22-05-2025
Is this the only test that fails? With your AddReads.jtr file, the crash now happened in the second step (-XX:AOTMode=create), whereas the original hs_err file was crashing in the first step (-XX:AOTMode=record). Is this crash intermittent? Can you provide instructions for reproducing the test environment so I can investigate?
22-05-2025
I don't have direct access to the machine. I can only submit patches to test and save the hs_err and .jtr files. The issue seems to be reproducible. I see it always happening.
22-05-2025
Reproduced without Compact Object Headers and attached AddReads.jtr.
22-05-2025
Can you build with debug symbols? There's no debug information in the hs_err file. Also, please run this test by itself and attach all log files created by the test. E.g., this test should create a log file with the name "add-reads-with-classpath.aotconfig.log".
22-05-2025
I added the add-reads-with-classpath.aotconfig.log . Debug symbols were enabled in our Alpine builds; but the crash is outside of our libs so this might cause the missing stack backtrace.
22-05-2025
hs_err file show that it hit issue with -XX:AOTMode=record . At this mode AOT code caching is not executed. Assigning this to [~iklam] and runtime. [~mdoerr] please also attach log file with UL output to see when it happens.
21-05-2025
[~kvn] or [~asmehra], can you have a look?
21-05-2025