JDK-8345147 : Crash in libsystem_malloc.dylib`nanov2_malloc_type (ESR_EC_DABORT_EL0)
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 24
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • OS: os_x
  • CPU: aarch64
  • Submitted: 2024-11-28
  • Updated: 2025-05-01
  • Resolved: 2025-05-01
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 25
25Resolved
Related Reports
Cloners :  
JDK-8355985 - Crash in libjvm.dylib`assert_is_oop_or_null() (ESR_EC_DABORT_EL0)
Duplicate :  
JDK-8350455 - Malloc errors when concurrently allocating and freeing memory.
Relates :  
JDK-8345500 - Timeout in jdk.jfr.api.consumer.streaming.TestJVMCrash
Relates :  
JDK-8345342 - Extend NMT, so that it can help catch memory corruptions
Relates :  
JDK-8344921 - jdk/internal/loader/ClassLoaderValue/ClassLoaderValueTest fails due to a JVM crash
Relates :  
JDK-8338883 - Show warning when CreateCoredumpOnCrash set, but core dump will not happen
Description
Test: jdk/jfr/api/consumer/streaming/TestOutOfProcessMigration.java

java.lang.RuntimeException: Process (pid = 14415) is no longer alive, exit value = 131

Core dump shows:

* thread #3, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0)
  * frame #0: 0x0000000196dcae60 libsystem_malloc.dylib`nanov2_malloc_type + 160
    frame #1: 0x0000000103d9c8ec libjvm.dylib`NMTPreInitAllocation::do_alloc(unsigned long) + 36
    frame #2: 0x0000000103dc2f5c libjvm.dylib`os::malloc(unsigned long, MemTag, NativeCallStack const&) + 196
    frame #3: 0x000000010367fda0 libjvm.dylib`AllocateHeap(unsigned long, MemTag, AllocFailStrategy::AllocFailEnum) + 80
    frame #4: 0x0000000103c90cc4 libjvm.dylib`LogOutputList::set_output_level(LogOutput*, LogLevel::type) + 80
    frame #5: 0x0000000103c8aacc libjvm.dylib`LogConfiguration::initialize(long) + 396
    frame #6: 0x0000000103f4eee8 libjvm.dylib`Threads::create_vm(JavaVMInitArgs*, bool*) + 172
    frame #7: 0x0000000103ac6c90 libjvm.dylib`JNI_CreateJavaVM + 104

Also as reported in JDK-8344921:

Test: jdk/internal/loader/ClassLoaderValue/ClassLoaderValueTest

* thread #38, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x600061ad0000)
  * frame #0: 0x00000001c4969470 libsystem_malloc.dylib`nanov2_allocate + 348
    frame #1: 0x00000001c49692b0 libsystem_malloc.dylib`nanov2_malloc + 64
    frame #2: 0x00000001c4986770 libsystem_malloc.dylib`_malloc_zone_malloc + 156
    frame #3: 0x00000001042dbeb4 libjvm.dylib`os::malloc(unsigned long, MemTag, NativeCallStack const&) + 460
    frame #4: 0x0000000103b98aac libjvm.dylib`AllocateHeap(unsigned long, MemTag, AllocFailStrategy::AllocFailEnum) + 80
    frame #5: 0x00000001041952cc libjvm.dylib`LightweightSynchronizer::inflate_into_object_header(oopDesc*, ObjectSynchronizer::InflateCause, JavaThread*, Thread*) + 296
Comments
Thank you for the note Jaikiran! This issue, after we removed the unrelated issue onto its own, does look like a dup of JDK-8350455.
01-05-2025
Hello Gerard, we have had several issues (including crashes) related to nanov2 implementation of malloc() on macos. For example there's https://bugs.openjdk.org/browse/JDK-8350455. We have filed a Apple feedback issue (mentioned in the comments of that other issue) but haven't officially heard back yet from Apple.
30-04-2025
Does anyone else have any thoughts here?
30-04-2025
Looking at the frame where the crash occurred: * thread #3, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0) * frame #0: 0x0000000196dcae60 libsystem_malloc.dylib`nanov2_malloc_type + 160 frame #1: 0x0000000103d9c8ec libjvm.dylib`NMTPreInitAllocation::do_alloc(unsigned long) + 36 It looks to be in libsystem_malloc.dylib, not hotspot. Could be a bug in macOS. We could of course also have double free here, but without a reproducable case, or more info (ex. core file) I'm not sure we can do much here for now.
30-04-2025
Created https://bugs.openjdk.org/browse/JDK-8355985 to track the Crash in libjvm.dylib`assert_is_oop_or_null() (ESR_EC_DABORT_EL0) as a separate issue. This one will track crash in "libsystem_malloc.dylib`nanov2_malloc_type"
30-04-2025
The crash seems to be in libsystem_malloc.dylib, which is not part of jdk, but macOS (system library). I think using ESR_EC_DABORT_EL0 as the pattern might not be helpful. Those look like very different crashes.
29-04-2025
ESR_EC_DABORT_EL0 is a data-abort ("segfault" for non-instruction load/store)
29-04-2025
The crash as reported in this issue: * thread #3, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0) * frame #0: 0x0000000196dcae60 libsystem_malloc.dylib`nanov2_malloc_type + 160 frame #1: 0x0000000103d9c8ec libjvm.dylib`NMTPreInitAllocation::do_alloc(unsigned long) + 36 comes from a different code altogether: * thread #39, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x1294f1bd8) * frame #0: 0x0000000104853f58 libjvm.dylib`assert_is_oop_or_null(zaddress) + 88 here we have a reasonable looking address value: 0x1294f1bd8.
29-04-2025
Darn, the latest links that David provided are gone.
29-04-2025
The crash happens in "assert_is_oop_or_null(zaddress)", where: bool oopDesc::is_oop_or_null(oop obj, bool ignore_mark_word) { return obj == nullptr ? true : is_oop(obj, ignore_mark_word); } bool oopDesc::is_oop(oop obj, bool ignore_mark_word) { if (!Universe::heap()->is_oop(obj)) { return false; } // Header verification: the mark is typically non-zero. If we're // at a safepoint, it must not be zero, except when using the new lightweight locking. // Outside of a safepoint, the header could be changing (for example, // another thread could be inflating a lock on this object). if (ignore_mark_word) { return true; } if (obj->mark().value() != 0) { return true; } return LockingMode == LM_LIGHTWEIGHT || !SafepointSynchronize::is_at_safepoint(); } bool CollectedHeap::is_oop(oop object) const { if (!is_object_aligned(object)) { return false; } if (!is_in(object)) { return false; } if (!klass_is_sane(object)) { return false; } return true; } template<typename T, typename A, ENABLE_IF(std::is_integral<T>::value)> constexpr bool is_aligned(T size, A alignment) { return (size & alignment_mask(alignment)) == 0; } static bool klass_is_sane(oop object) { if (UseCompactObjectHeaders) { // With compact headers, we can't safely access the Klass* when // the object has been forwarded, because non-full-GC-forwarding // temporarily overwrites the mark-word, and thus the Klass*, with // the forwarding pointer, and here we have no way to make a // distinction between Full-GC and regular GC forwarding. markWord mark = object->mark(); if (mark.is_forwarded()) { // We can't access the Klass*. We optimistically assume that // it is ok. This happens very rarely. return true; } return Metaspace::contains(mark.klass_without_asserts()); } return Metaspace::contains(object->klass_without_asserts()); } markWord oopDesc::mark() const { return Atomic::load(&_mark); } Only obj->mark().value() seem to dereference the value, so somewhere where we do mark.value()? Can obj->mark() ever fail? (for example if it's regular int, as opposed to std::atomic<int> ?
29-04-2025
Test: vmTestbase/gc/gctests/StringInternSync2/TestDescription.java frame #12: 0x0000000115f5d3b8 * thread #39, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x1294f1bd8) * frame #0: 0x0000000104853f58 libjvm.dylib`assert_is_oop_or_null(zaddress) + 88 frame #1: 0x000000010454343c libjvm.dylib`AccessInternal::PostRuntimeDispatch<ZBarrierSet::AccessBarrier<593988ull, ZBarrierSet>, (AccessInternal::BarrierType)2, 593988ull>::oop_access_barrier(void*) + 92 frame #2: 0x0000000105299f64 libjvm.dylib`StringTableLookupOop::equals(WeakHandle*) + 92 frame #3: 0x0000000105299e34 libjvm.dylib`ConcurrentHashTable<StringTableConfig, (MemTag)11>::Node* ConcurrentHashTable<StringTableConfig, (MemTag)11>::get_node<StringTableLookupOop>(ConcurrentHashTable<StringTableConfig, (MemTag)11>::Bucket const*, StringTableLookupOop&, bool*, unsigned long*) const + 128 frame #4: 0x000000010529ab28 libjvm.dylib`bool ConcurrentHashTable<StringTableConfig, (MemTag)11>::internal_insert_get<StringTableLookupOop, bool ConcurrentHashTable<StringTableConfig, (MemTag)11>::insert<StringTableLookupOop>(Thread*, StringTableLookupOop&, WeakHandle const&, bool*, bool*)::NOP>(Thread*, StringTableLookupOop&, WeakHandle const&, bool ConcurrentHashTable<StringTableConfig, (MemTag)11>::insert<StringTableLookupOop>(Thread*, StringTableLookupOop&, WeakHandle const&, bool*, bool*)::NOP&, bool*, bool*) + 368 frame #5: 0x0000000105296034 libjvm.dylib`StringTable::do_intern(StringWrapperInternal const&, unsigned long, JavaThread*) + 580 frame #6: 0x00000001052959d4 libjvm.dylib`StringTable::intern(StringWrapperInternal const&, JavaThread*) + 488 frame #7: 0x0000000105295c44 libjvm.dylib`StringTable::intern(oop, JavaThread*) + 444 frame #8: 0x0000000104ce6320 libjvm.dylib`JVM_InternString + 344 frame #9: 0x0000000115f5d3b8 frame #12: 0x0000000115f5d3b8 thread #41, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x600026f40000) frame #0: 0x00000001afb59470 libsystem_malloc.dylib`nanov2_allocate + 348 frame #1: 0x00000001afb592b0 libsystem_malloc.dylib`nanov2_malloc + 64 frame #2: 0x00000001afb76770 libsystem_malloc.dylib`_malloc_zone_malloc + 156 frame #3: 0x0000000105109c48 libjvm.dylib`os::malloc(unsigned long, MemTag, NativeCallStack const&) + 184 frame #4: 0x00000001044e8ff8 libjvm.dylib`AllocateHeap(unsigned long, MemTag, AllocFailStrategy::AllocFailEnum) + 124 frame #5: 0x000000010529aa1c libjvm.dylib`bool ConcurrentHashTable<StringTableConfig, (MemTag)11>::internal_insert_get<StringTableLookupOop, bool ConcurrentHashTable<StringTableConfig, (MemTag)11>::insert<StringTableLookupOop>(Thread*, StringTableLookupOop&, WeakHandle const&, bool*, bool*)::NOP>(Thread*, StringTableLookupOop&, WeakHandle const&, bool ConcurrentHashTable<StringTableConfig, (MemTag)11>::insert<StringTableLookupOop>(Thread*, StringTableLookupOop&, WeakHandle const&, bool*, bool*)::NOP&, bool*, bool*) + 100 frame #6: 0x0000000105296034 libjvm.dylib`StringTable::do_intern(StringWrapperInternal const&, unsigned long, JavaThread*) + 580 frame #7: 0x00000001052959d4 libjvm.dylib`StringTable::intern(StringWrapperInternal const&, JavaThread*) + 488 frame #8: 0x0000000105295c44 libjvm.dylib`StringTable::intern(oop, JavaThread*) + 444 frame #9: 0x0000000104ce6320 libjvm.dylib`JVM_InternString + 344 frame #10: 0x0000000115f5d3b8 thread #44, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x600022f50000) frame #0: 0x00000001afb59470 libsystem_malloc.dylib`nanov2_allocate + 348 frame #1: 0x00000001afb592b0 libsystem_malloc.dylib`nanov2_malloc + 64 frame #2: 0x00000001afb76770 libsystem_malloc.dylib`_malloc_zone_malloc + 156 frame #3: 0x0000000105109c48 libjvm.dylib`os::malloc(unsigned long, MemTag, NativeCallStack const&) + 184 frame #4: 0x00000001044e8ff8 libjvm.dylib`AllocateHeap(unsigned long, MemTag, AllocFailStrategy::AllocFailEnum) + 124 frame #5: 0x000000010529aa1c libjvm.dylib`bool ConcurrentHashTable<StringTableConfig, (MemTag)11>::internal_insert_get<StringTableLookupOop, bool ConcurrentHashTable<StringTableConfig, (MemTag)11>::insert<StringTableLookupOop>(Thread*, StringTableLookupOop&, WeakHandle const&, bool*, bool*)::NOP>(Thread*, StringTableLookupOop&, WeakHandle const&, bool ConcurrentHashTable<StringTableConfig, (MemTag)11>::insert<StringTableLookupOop>(Thread*, StringTableLookupOop&, WeakHandle const&, bool*, bool*)::NOP&, bool*, bool*) + 100 frame #6: 0x0000000105296034 libjvm.dylib`StringTable::do_intern(StringWrapperInternal const&, unsigned long, JavaThread*) + 580 frame #7: 0x00000001052959d4 libjvm.dylib`StringTable::intern(StringWrapperInternal const&, JavaThread*) + 488 frame #8: 0x0000000105295c44 libjvm.dylib`StringTable::intern(oop, JavaThread*) + 444 frame #9: 0x0000000104ce6320 libjvm.dylib`JVM_InternString + 344 frame #10: 0x0000000115f5d3b8
19-01-2025
> there are many cases of the test failing that way, without a core or any sign of a kerne crash in lldb. [~dholmes] Yes, those are tracked by JDK-8345500, and I will try to look into those as well.
20-12-2024
[~gziemski] there are many cases of the test failing that way, without a core or any sign of a kerne crash in lldb.
19-12-2024
So far I get crashes and I see: java.lang.RuntimeException: Process (pid = 14415) is no longer alive, exit value = 131 but no core files (and exception type is SIGQUIT, not ESR_EC_DABORT_EL0).
19-12-2024
To anyone adding a sighting of this failure - could you kindly report the macOS version as well? (look for either "javatestOS" or "testJDK_os.version")
18-12-2024
This latest failure took place on javatestOS=Mac OS X 15.0.1 (aarch64), so the crashes are still restricted to 15.0.1 for now.
16-12-2024
I re-organized the CreateCoredumpOnCrash handling code for JDK-8338883 and will now print out a warning if the feature is set, but the os does not allow writing the core files. The core files do get created in these tests, so it can't be that. However it all happens early in the process of starting VM, so I am wondering if we pull in some dynamic library now, that otherwise we didn't.
09-12-2024
I ran the test continuously 200 times (M1 macOS 15.1.1) and other than the timeouts seen in https://bugs.openjdk.org/browse/JDK-8345500 I do not get these specific crashes.
05-12-2024
Another case: * thread #3, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0) * frame #0: 0x0000000182e86e38 dyld`dyld3::MachOFile::trieWalk(Diagnostics&, unsigned char const*, unsigned char const*, char const*) + 312 frame #1: 0x0000000182e55664 dyld`dyld4::Loader::hasExportedSymbol(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::ExportedSymbolMode, dyld4::Loader::ResolverMode, dyld4::Loader::ResolvedSymbol*, dyld3::Array<dyld4::Loader const*>*) const + 388 frame #2: 0x0000000182e6d5dc dyld`invocation function for block in dyld4::APIs::dlsym(void*, char const*) + 232 frame #3: 0x0000000182e44404 dyld`dyld4::RuntimeLocks::withLoadersReadLock(void () block_pointer) + 100 frame #4: 0x0000000182e6d058 dyld`dyld4::APIs::dlsym(void*, char const*) + 984 frame #5: 0x00000001067eb360 libjvm.dylib`os::Posix::init() + 32 frame #6: 0x00000001067e5c90 libjvm.dylib`os::init() + 240 frame #7: 0x000000010696a694 libjvm.dylib`Threads::create_vm(JavaVMInitArgs*, bool*) + 120 frame #8: 0x00000001064e1f9c libjvm.dylib`JNI_CreateJavaVM + 104 frame #9: 0x0000000104ec6438 libjli.dylib`JavaMain + 256 frame #10: 0x0000000104ec972c libjli.dylib`ThreadJavaMain + 12 frame #11: 0x00000001831bf2e4 libsystem_pthread.dylib`_pthread_start + 136
04-12-2024
So far, running on my personal M1 with macOS 15.1.1 all I see is the same timeouts as in https://bugs.openjdk.org/browse/JDK-8345500 and the terminations come from cleaning out timed-out tests, not unexpected crashes that dump core files.
04-12-2024
Filed JDK-8345500 to track the timeouts I found trying to replicate this issue. It looks possibly related, but happens on macOS 14.7 as opposed to 15.0 as it is here, and is not a crash, but timeout, though the stack traces look similar where here we crash early in startup, ex: * thread #1, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0) * frame #0: 0x0000000192a5cc0c dyld`DyldSharedCache::getIndexedImageEntry(unsigned int, unsigned long long&, unsigned long long&) const + 56 frame #1: 0x0000000192a20b80 dyld`dyld4::RuntimeState::cachedDylibLoadAddress(unsigned short) const + 52 frame #2: 0x0000000192a23f7c dyld`dyld4::RuntimeState::buildInterposingTables() + 260 frame #3: 0x0000000192a192f8 dyld`dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 1780 frame #4: 0x0000000192a18bc0 dyld`dyld4::start(dyld4::KernelArgs*, void*, void*)::$_0::operator()() const + 544 frame #5: 0x0000000192a1805c dyld`start + 2304 and in the other one we see just: * thread #1 * frame #0: 0x0000000104cb8b70 dyld`_dyld_start Could be related, especially that all of those are in jdk.jfr.api.consumer.streaming
04-12-2024
I went through several commits from the current jdk, but nothing jumped out at me. Time to see how we can turn ON debug malloc in MACH5 ?
03-12-2024
ESR_EC_DABORT_EL0 means: Exception-Syndrome-Register Exception-Class Data-Abort EL0
03-12-2024
Test: jdk/jfr/api/consumer/streaming/TestOutOfProcessMigration.java ---------------------------------------- (lldb) target create "/System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-24+27-3445/macosx-aarch64.jdk/jdk-24.jdk/Contents/Home/bin/java" --core "/System/Volumes/Data/mesos/work_dir/slaves/8bc4eeb1-8ff8-46bb-a0a1-2fa14e249a7b-S61/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/07a517a6-21a5-4708-9c58-b7ef6e5b6933/runs/bf0e43e1-db51-4b18-bcc8-3d11ebc4075c/testoutput/test-support/jtreg_open_test_jdk_tier3/jdk/jfr/api/consumer/streaming/TestOutOfProcessMigration/core.40303" Core file '/System/Volumes/Data/mesos/work_dir/slaves/8bc4eeb1-8ff8-46bb-a0a1-2fa14e249a7b-S61/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/07a517a6-21a5-4708-9c58-b7ef6e5b6933/runs/bf0e43e1-db51-4b18-bcc8-3d11ebc4075c/testoutput/test-support/jtreg_open_test_jdk_tier3/jdk/jfr/api/consumer/streaming/TestOutOfProcessMigration/core.40303' (arm64) was loaded. (lldb) thread backtrace all thread #1 frame #0: 0x0000000192d59e34 libsystem_kernel.dylib`mach_msg2_trap + 8 frame #1: 0x0000000192d6c5d0 libsystem_kernel.dylib`mach_msg2_internal + 80 frame #2: 0x0000000192d629d8 libsystem_kernel.dylib`mach_msg_overwrite + 480 frame #3: 0x0000000192d5a17c libsystem_kernel.dylib`mach_msg + 24 frame #4: 0x0000000192e825bc CoreFoundation`__CFRunLoopServiceMachPort + 160 frame #5: 0x0000000192e80e6c CoreFoundation`__CFRunLoopRun + 1212 frame #6: 0x0000000192e80334 CoreFoundation`CFRunLoopRunSpecific + 572 frame #7: 0x0000000102354f64 libjli.dylib`CreateExecutionEnvironment + 412 frame #8: 0x0000000102350a0c libjli.dylib`JLI_Launch + 560 frame #9: 0x000000010230bbbc java`main + 396 frame #10: 0x0000000192a18274 dyld`start + 2840 * thread #2, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x1f7244068) * frame #0: 0x0000000192a2fce4 dyld`invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 940 frame #1: 0x0000000192a6e188 dyld`invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 172 frame #2: 0x0000000192a61b24 dyld`invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 496 frame #3: 0x0000000192a142dc dyld`dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 300 frame #4: 0x0000000192a60ab8 dyld`dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192 frame #5: 0x0000000192a6348c dyld`dyld3::MachOFile::forEachInitializerPointerSection(Diagnostics&, void (unsigned int, unsigned int, bool&) block_pointer) const + 160 frame #6: 0x0000000192a6de7c dyld`dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 432 frame #7: 0x0000000192a2f8ac dyld`dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 176 frame #8: 0x0000000192a36e88 dyld`dyld4::JustInTimeLoader::runInitializers(dyld4::RuntimeState&) const + 36 frame #9: 0x0000000192a2ff68 dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&, dyld3::Array<dyld4::Loader const*>&) const + 312 frame #10: 0x0000000192a34258 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_0::operator()() const + 180 frame #11: 0x0000000192a30158 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 412 frame #12: 0x0000000192a48488 dyld`dyld4::APIs::dlopen_from(char const*, int, void*) + 2432 frame #13: 0x000000010235545c libjli.dylib`LoadJavaVM + 180 frame #14: 0x0000000102350a50 libjli.dylib`JLI_Launch + 628 frame #15: 0x000000010230bbbc java`main + 396 frame #16: 0x0000000102355fc0 libjli.dylib`apple_main + 88 frame #17: 0x0000000192d9b2e4 libsystem_pthread.dylib`_pthread_start + 136 (lldb) quit
03-12-2024
Test: jdk/jfr/api/consumer/streaming/TestJVMCrash.java ---------------------------------------- [2024-12-03 17:11:58] [/System/Volumes/Data/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-macosx/Xcode14.3.1+1.0/devkit-macosx-Xcode14.3.1+1.0.tar.gz/Xcode/Contents/Developer/usr/bin/lldb, --core, /System/Volumes/Data/mesos/work_dir/slaves/8bc4eeb1-8ff8-46bb-a0a1-2fa14e249a7b-S61/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/07a517a6-21a5-4708-9c58-b7ef6e5b6933/runs/bf0e43e1-db51-4b18-bcc8-3d11ebc4075c/testoutput/test-support/jtreg_open_test_jdk_tier3/jdk/jfr/api/consumer/streaming/TestJVMCrash/core.40924, /System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-24+27-3445/macosx-aarch64.jdk/jdk-24.jdk/Contents/Home/bin/java, -o, thread backtrace all, -o, quit] timeout=120000 in /System/Volumes/Data/mesos/work_dir/slaves/8bc4eeb1-8ff8-46bb-a0a1-2fa14e249a7b-S61/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/07a517a6-21a5-4708-9c58-b7ef6e5b6933/runs/bf0e43e1-db51-4b18-bcc8-3d11ebc4075c/testoutput/test-support/jtreg_open_test_jdk_tier3/jdk/jfr/api/consumer/streaming/TestJVMCrash ---------------------------------------- (lldb) target create "/System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-24+27-3445/macosx-aarch64.jdk/jdk-24.jdk/Contents/Home/bin/java" --core "/System/Volumes/Data/mesos/work_dir/slaves/8bc4eeb1-8ff8-46bb-a0a1-2fa14e249a7b-S61/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/07a517a6-21a5-4708-9c58-b7ef6e5b6933/runs/bf0e43e1-db51-4b18-bcc8-3d11ebc4075c/testoutput/test-support/jtreg_open_test_jdk_tier3/jdk/jfr/api/consumer/streaming/TestJVMCrash/core.40924" Core file '/System/Volumes/Data/mesos/work_dir/slaves/8bc4eeb1-8ff8-46bb-a0a1-2fa14e249a7b-S61/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/07a517a6-21a5-4708-9c58-b7ef6e5b6933/runs/bf0e43e1-db51-4b18-bcc8-3d11ebc4075c/testoutput/test-support/jtreg_open_test_jdk_tier3/jdk/jfr/api/consumer/streaming/TestJVMCrash/core.40924' (arm64) was loaded. (lldb) thread backtrace all * thread #1, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0) * frame #0: 0x0000000192a5cc0c dyld`DyldSharedCache::getIndexedImageEntry(unsigned int, unsigned long long&, unsigned long long&) const + 56 frame #1: 0x0000000192a20b80 dyld`dyld4::RuntimeState::cachedDylibLoadAddress(unsigned short) const + 52 frame #2: 0x0000000192a23f7c dyld`dyld4::RuntimeState::buildInterposingTables() + 260 frame #3: 0x0000000192a192f8 dyld`dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 1780 frame #4: 0x0000000192a18bc0 dyld`dyld4::start(dyld4::KernelArgs*, void*, void*)::$_0::operator()() const + 544 frame #5: 0x0000000192a1805c dyld`start + 2304 (lldb) quit
03-12-2024
Now crashing in places other than malloc, but the same JFR tests are involved.
03-12-2024
# git log --oneline src/hotspot/ 3d0d0e62900 8345012: os::build_agent_function_name potentially wastes a byte when allocating the buffer 7c944ee6f4d 8345172: x86: Some CPU feature asserts are declared as 32-bit only 1ca764454b1 8339480: Build static-jdk image with a statically linked launcher b8233989e76 8345267: Fix memory leak in JVMCIEnv dtor 0b0f83c01e3 8345220: Serial: Refactor TenuredGeneration::promotion_attempt_is_safe dfcbfb5a410 8345236: RISC-V: Remove revb_h_h_u and revb_h_w_u macro assembler routines f5ebda43709 8345173: BlockLocationPrinter::print_location misses a ResourceMark 28ae281b42c 8337511: Implement JEP 404: Generational Shenandoah (Experimental) 6bea1b6cf1f 8344727: [JVMCI] Export the CompileBroker compilation activity mode for Truffle compiler control 095e769f476 8345237: 32-bit Zero builds fail with assert(has_klass_gap()) failed: precondition 08d563ba150 8345110: RISC-V: Optimize and and clean up byte reverse assembler routines 7dc00d39b4e 8345154: IGV: Show Parse and Assertion Predicate type as extra label 1e086b1d730 8340103: Add internal set_flag function to VMATree edfe28541a6 8344306: RISC-V: Add zicond d33ad07c32f 8334493: Remove SecurityManager Permissions infrastructure from DiagnosticCommands e096660a189 8345043: [ASAN] methodMatcher.cpp report reading from a region of size 0 [-Werror=stringop-overread] 1a07d542ec8 8343703: Symbol name cleanups after JEP 479 a0df0a527fc 8340731: Cleanup remaining IA64 references in hotspot code f51363e0277 8344913: Improve -Xlog:cds+map+oop logging for Java mirrors f6d29909bb5 8344824: CDS dump crashes when member_method of a lambda proxy is null 15378a78535 8345126: [BACKOUT] JDK-8318127: align_up has potential overflow 494806286f1 8318127: align_up has potential overflow 461ffafeba4 8331735: UpcallLinker::on_exit races with GC when copying frame anchor 8343144: UpcallLinker::on_entry racingly clears pending exception with GC safepoin ts 8286875: ProgrammableUpcallHandler::on_entry/on_exit access thread fields from native eb0d1ce9487 8344355: Register corruption in MacroAssembler::lookup_secondary_supers_table_var: x86-64 only 82137db24da 8345047: RISC-V: Remove explicit use of AvoidUnalignedAccesses in interpreter 1f6144ef260 8345050: Fix -Wzero-as-null-pointer warning in MemPointer ctor 3689f3909ee 8344013: "bad tag in log" assert with +LogCompilation +CITimeVerbose d752f19611f 8343427: Class file load hook crashes on archived classes from multi-release JARs 0054bbed7fc 8343756: CAN_SHOW_REGISTERS_ON_ASSERT for Windows 9793e73bc1b 8344853: Parallel: Improve comments in psParallelCompact 57d35f98f69 8344382: RISC-V: CASandCAEwithNegExpected fails with Zacas 01052035751 8337660: C2: basic blocks with only BoxLock nodes are wrongly treated as empty 5e0d42b6a63 8344916: RISC-V: Misaligned access in array fill stub 8de158aefe6 8339134: Callers of Exceptions::fthrow should ensure exception message lengths avoid the INT_MAX limits of os::vsnprintf 08dfc4a42e5 8344213: Cleanup OpaqueLoop*Node verification code for Assertion Predicates 133419177d8 8334474: RISC-V: verify perf of ExpandBits/CompressBits (rvv) cb1c73663e9 8344363: FullGCForwarding::initialize_flags is called after ObjLayout::initialize a83cfe26583 8344917: Fix recent NULL usage backsliding 822a1554cb0 8341427: JFR: Adjust object sampler span handling 98b667834c4 8343741: SA jstack --mixed should print information about VM locks 079f503d962 8344568: Renaming ceil_log2 to log2i_ceil 82c3612d775 8344830: [BACKOUT] JDK-8341334: CDS: Parallel relocation 2ea0364b6e3 8343893: Test jdk/jfr/event/runtime/TestNativeMemoryUsageEvents.java failed: heap should have grown and NMT should show that: expected 0 > 0 db44e97c5df 8344798: Shenandoah: Use more descriptive variable names in shPhaseTimings.cpp 93d4ad4dd4c 8344763: cpCache print_on doesn't handle nulls dc9a6ef6100 8341137: Optimize long vector multiplication using x86 VPMUL[U]DQ instruction 8f22db23a50 8330606: Redefinition doesn't but should verify the new klass 93aa7e2fcf8 8328085: C2: Use after free in PhaseChaitin::Register_Allocate() 4fbf272017d (tag: jdk-24+25) 8344526: RISC-V: implement -XX:+VerifyActivationFrameSize
03-12-2024
ILW = HLM = P3 W/A: disable NMT
03-12-2024
I will try and see if there any macOS specific changes recently, that could explain this regression...
02-12-2024
Trying the example from "man libgmalloc" shows that the "DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib" mechanism still works on current macOS, so we should try and enable it for our testing to see if it flushes anything out.
02-12-2024
We could try turning ON some of the macOS malloc debugging mechanisms: - man libgmalloc - https://developer.apple.com/library/archive/documentation/Performance/Conceptual/ManagingMemory/Articles/MallocDebug.html I will see if they are still useable...
02-12-2024
This is quite strange. Both of these crash inside of the libc malloc allocator, indicating that we've messed up the state of malloc or that malloc has a bug. The issue probably occurred much earlier and our stack traces only represent when the state inconsistency actually caused a crash. An interesting insight here is that in the first stack trace we are in NMTPreInit and performing an allocation. When we are in NMTPreInit we do *not* have malloc headers, and so NMT does not touch the memory that malloc returns to the VM. This indicates that it's not something that NMT is doing which is causing that particular crash.
28-11-2024