JDK-8273505 : runtime/cds/appcds/loaderConstraints/DynamicLoaderConstraintsTest.java#default-cl crashed with SIGSEGV in MetaspaceShared::link_shared_classes
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 18
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: os_x
  • CPU: x86_64
  • Submitted: 2021-09-08
  • Updated: 2022-06-02
  • Resolved: 2021-09-20
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 17 JDK 18
17.0.2Fixed 18 b16Fixed
Related Reports
Relates :  
JDK-8273109 - runtime/cds/appcds/loaderConstraints/DynamicLoaderConstraintsTest times out
Relates :  
JDK-8274753 - ZGC: SEGV in MetaspaceShared::link_shared_classes
Description
The following test crashed in the JDK18 CI:

runtime/cds/appcds/loaderConstraints/DynamicLoaderConstraintsTest.java#default-cl

Here's a snippet from the log file:

[148.932s][info][class,loader,constraints] purging loader MyClassLoader @1ddc4ec2 from constraint for name com/sun/net/httpserver/HttpExchange
[148.932s][info][class,loader,constraints] new loader list:
[148.932s][info][class,loader,constraints]     [0]: 'platform'
[148.932s][info][class,loader,constraints] purging complete constraint for name com/sun/net/httpserver/HttpExchange
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000110d51227, pid=89224, tid=8963
#
# JRE version: Java(TM) SE Runtime Environment (18.0+14) (fastdebug build 18-ea+14-700)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 18-ea+14-700, compiled mode, sharing, tiered, compressed class ptrs, g1 gc, bsd-amd64)
# Problematic frame:
# V  [libjvm.dylib+0xd51227]  MetaspaceShared::link_shared_classes(JavaThread*)+0x1d7
#
# Core dump will be written. Default location: core.89224
#
# An error report file with more information is saved as:
# /mesos/work_dir/slaves/369e65c9-433d-40ff-b6a0-b4e65701097a-S4971/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/ab5006d0-0738-44c0-80b4-bf4d6f6ec0b2/runs/887fb39d-d2e0-4453-86af-e0b2edea301c/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/scratch/0/hs_err_pid89224.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
];
 stderr: []
 exitValue = 134

java.lang.RuntimeException: Expected to get exit value of [0]

	at jdk.test.lib.process.OutputAnalyzer.shouldHaveExitValue(OutputAnalyzer.java:489)
	at jdk.test.lib.cds.CDSTestUtils$Result.assertNormalExit(CDSTestUtils.java:195)
	at DynamicLoaderConstraintsTest.doTest(DynamicLoaderConstraintsTest.java:140)
	at DynamicLoaderConstraintsTest.doTest(DynamicLoaderConstraintsTest.java:98)
	at DynamicArchiveTestBase.runTest(DynamicArchiveTestBase.java:74)
	at DynamicLoaderConstraintsTest.main(DynamicLoaderConstraintsTest.java:89)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
	at java.base/java.lang.Thread.run(Thread.java:833)

JavaTest Message: Test threw exception: java.lang.RuntimeException: Expected to get exit value of [0]

JavaTest Message: shutting down test

STATUS:Failed.`main' threw exception: java.lang.RuntimeException: Expected to get exit value of [0]
----------rerun:(49/11225)*----------


Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x00007fe665daf220):  JavaThread "DestroyJavaVM" [_thread_in_vm, id=8963, stack(0x0000000111986000,0x0000000111a86000)]

Stack: [0x0000000111986000,0x0000000111a86000],  sp=0x0000000111a85bc0,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.dylib+0xd51227]  MetaspaceShared::link_shared_classes(JavaThread*)+0x1d7
V  [libjvm.dylib+0x67813c]  DynamicArchive::prepare_for_dynamic_dumping()+0x3c
V  [libjvm.dylib+0x104636a]  JavaThread::invoke_shutdown_hooks()+0x3a
V  [libjvm.dylib+0x104659d]  Threads::destroy_vm()+0x1bd
V  [libjvm.dylib+0x970ae3]  jni_DestroyJavaVM+0x163
C  [libjli.dylib+0x51c0]  JavaMain+0xc20
C  [libjli.dylib+0x7649]  ThreadJavaMain+0x9
C  [libsystem_pthread.dylib+0x3661]  _pthread_body+0x154
C  [libsystem_pthread.dylib+0x350d]  _pthread_body+0x0
C  [libsystem_pthread.dylib+0x2bf9]  thread_start+0xd


siginfo: si_signo: 11 (SIGSEGV), si_code: 0 (unknown), si_addr: 0x0000000000000000
Comments
Fix Request (17u) Backporting this bug fix would fix the above issue and also simplify backporting the fixes for JDK-8274935 and JDK-8274753. The patch is small and applied cleanly except that function MetaspaceShared::link_shared_classes() is called MetaspaceShared::link_and_cleanup_shared_classes() in JDK-17u. The backport was tested with Mach5 tiers 1-2 on Linux, Mac OS, and Windows, and Mach5 tiers 3-5 on Linux x64. Review of the fix: https://github.com/openjdk/jdk17u/pull/187
13-10-2021
Changeset: a67f0f9d Author: Calvin Cheung <ccheung@openjdk.org> Date: 2021-09-20 16:56:32 +0000 URL: https://git.openjdk.java.net/jdk/commit/a67f0f9dc04200c22db05e277346f24d31306b53
20-09-2021
(lldb) p *collect_cld._loaded_cld._data[110] (ClassLoaderData) $4 = { _holder = { _obj = 0xbabababababababa } _class_loader = { _obj = 0xabababababababab } _metaspace = 0x001cabababababab _metaspace_lock = 0xabababababababab _unloading = true _has_class_mirror_holder = true _modified_oops = true _keep_alive = -1414812757 _claim = 64 _handles = { _head = 0x0000000000000000 } _dependency_count = -1162167622 _klasses = 0xbabababababababa _packages = 0xbabababababababa _modules = 0xbabababababababa _unnamed_module = 0xbabababababababa _dictionary = 0xbabababababababa _jmethod_ids = 0xbabababababababa _deallocate_list = 0xbabababababababa _next = 0xabababababababab _class_loader_klass = 0x0023abababababab _name = 0xabababababababab _name_and_id = 0xabababababababab _trace_id = 64 } The cld was unloaded.
14-09-2021
I'm looking at this again. If there's an inc_keep_alive() I don't know why the class loader is being unloaded.
14-09-2021
ILW = HLM = P3
14-09-2021
[~coleenp] Do you mean the following? diff --git a/src/hotspot/share/cds/metaspaceShared.cpp b/src/hotspot/share/cds/metaspaceShared.cpp index a67450cb122..4b77f495636 100644 --- a/src/hotspot/share/cds/metaspaceShared.cpp +++ b/src/hotspot/share/cds/metaspaceShared.cpp @@ -584,6 +584,7 @@ void VM_PopulateDumpSharedSpace::doit() { class CollectCLDClosure : public CLDClosure { GrowableArray<ClassLoaderData*> _loaded_cld; + GrowableArray<Handle> _loaded_cld_handles; // keep the CLDs alive public: CollectCLDClosure() {} ~CollectCLDClosure() { @@ -596,6 +597,7 @@ public: if (!cld->is_unloading()) { cld->inc_keep_alive(); _loaded_cld.append(cld); + _loaded_cld_handles.append(Handle(Thread::current(), cld->holder_phantom())); } }
10-09-2021
Oh yeah, that doesn't seem safe. The classes are kept alive during the CLDG iteration but anything can happen afterwards, and since link_class_for_cds can safepoint, we can't really put an NSV around so that there isn't a handshake making cld memory go away. You could try creating a sibling growable array of Handle(cld->holder()), that keeps the CLD alive while this is linking it.
09-09-2021
I think the problem may be that the cld has been garbage collected. [~coleenp] any suggestions?
09-09-2021
Initial core dump analysis using lldb: call stack: (lldb) bt * thread #4, stop reason = signal SIGSTOP frame #0: 0x00007fff682bcb66 libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x00007fff68487080 libsystem_pthread.dylib`pthread_kill + 333 frame #2: 0x00007fff682181ae libsystem_c.dylib`abort + 127 frame #3: 0x0000000110e11c21 libjvm.dylib`os::abort(dump_core=true, siginfo=<unavailable>, context=<unavailable>) at os_posix.cpp:1971:5 [opt] frame #4: 0x00000001110e78ec libjvm.dylib`VMError::report_and_die(id=11, message=<unavailable>, detail_fmt="%s", detail_args=0x0000000111a85700, thread=0x00007fe665daf220, pc="D\x8bs\fE\x85?A??H\x8b\x03H\x89??\x90\x10\x01", siginfo=0x0000000111a85aa0, context=0x0000000111a85b08, filename=0x0000000000000000, lineno=0, size=0) at vmError.cpp:1646:7 [opt] frame #5: 0x00000001110e6cf5 libjvm.dylib`VMError::report_and_die(thread=<unavailable>, sig=<unavailable>, pc=<unavailable>, siginfo=<unavailable>, context=<unavailable>, detail_fmt=<unavailable>) at vmError.cpp:1320:3 [opt] frame #6: 0x00000001110e7a41 libjvm.dylib`VMError::report_and_die(thread=<unavailable>, sig=<unavailable>, pc=<unavailable>, siginfo=<unavailable>, context=<unavailable>) at vmError.cpp:1326:3 [opt] frame #7: 0x0000000110f2af45 libjvm.dylib`::JVM_handle_bsd_signal(sig=11, info=0x0000000111a85aa0, ucVoid=0x0000000111a85b08, abort_if_unrecognized=1) at signals_posix.cpp:661:5 [opt] frame #8: 0x00007fff6847af5a libsystem_platform.dylib`_sigtramp + 26 frame #9: 0x0000000110d51228 libjvm.dylib`MetaspaceShared::link_shared_classes(JavaThread*) [inlined] Klass::layout_helper(this=0xbabababababababa) const at klass.hpp:273:49 [opt] frame #10: 0x0000000110d51227 libjvm.dylib`MetaspaceShared::link_shared_classes(JavaThread*) [inlined] Klass::is_instance_klass(this=0xbabababababababa) const at klass.hpp:598 [opt] * frame #11: 0x0000000110d51227 libjvm.dylib`MetaspaceShared::link_shared_classes(__the_thread__=<unavailable>) at metaspaceShared.cpp:663 [opt] frame #12: 0x000000011067813c libjvm.dylib`DynamicArchive::prepare_for_dynamic_dumping() at dynamicArchive.cpp:358:3 [opt] frame #13: 0x000000011104636a libjvm.dylib`JavaThread::invoke_shutdown_hooks(this=0x00007fe665daf220) at thread.cpp:3282:5 [opt] frame #14: 0x000000011104659d libjvm.dylib`Threads::destroy_vm() at thread.cpp:3368:11 [opt] frame #15: 0x0000000110970ae3 libjvm.dylib`::jni_DestroyJavaVM(JavaVM *) [inlined] jni_DestroyJavaVM_inner(vm=<unavailable>) at jni.cpp:3758:3 [opt] frame #16: 0x0000000110970984 libjvm.dylib`::jni_DestroyJavaVM(vm=<unavailable>) at jni.cpp:3770 [opt] frame #17: 0x000000010df641c0 libjli.dylib`JavaMain(_args=<unavailable>) at java.c:0 [opt] frame #18: 0x000000010df66649 libjli.dylib`ThreadJavaMain(args=<unavailable>) at java_md_macosx.m:722:29 [opt] frame #19: 0x00007fff68484661 libsystem_pthread.dylib`_pthread_body + 340 frame #20: 0x00007fff6848450d libsystem_pthread.dylib`_pthread_start + 377 frame #21: 0x00007fff68483bf9 libsystem_pthread.dylib`thread_start + 13 (lldb) f 11 frame #11: 0x0000000110d51227 libjvm.dylib`MetaspaceShared::link_shared_classes(__the_thread__=<unavailable>) at metaspaceShared.cpp:663 [opt] (lldb) fr v (JavaThread *) __the_thread__ = <variable not available> (ResourceMark) rm = { _impl = { _area = 0x00007fe6633d82a0 _saved_state = { _chunk = 0x00007fe660800020 _hwm = 0x00007fe660800038 "??b^ _max = 0x00007fe660800410 "????????????????4453-86af-e0b2edea301c/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/classes/4/test/hotspot/jtreg/runtime/cds/appcds:/mesos/work_dir/jib-master/install/jdk-18+14-700/src.full/open/test/hotspot/jtreg/runtime/cds/appcds:/mesos/work_dir/slaves/369e65c9-433d-40ff-b6a0-b4e65701097a-S4971/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/ab5006d0-0738-44c0-80b4-bf4d6f6ec0b2/runs/887fb39d-d2e0-4453-86af-e0b2edea301c/testoutput/test-support/jtreg_" _size_in_bytes = 984 _nesting = 1 } } _thread = 0x00007fe665daf220 _previous_resource_mark = 0x0000000111a85ca0 } (CollectCLDClosure) collect_cld = { _loaded_cld = { GrowableArrayWithAllocator<ClassLoaderData *, GrowableArray<ClassLoaderData *> > = { GrowableArrayView<ClassLoaderData *> = { GrowableArrayBase = { ResourceObj = { _allocation_t = ([0] = 18446744069118338023, [1] = 0) } _len = 131 _max = 256 } _data = 0x00007fe65ecd7638 } } _metadata = { _bits = 0 _nesting_check = (_nesting = 2) } } } (bool) has_linked = false (int) i = 110 (ClassLoaderData *) cld = <variable not available> (Klass *) klass = 0xbabababababababa So it looks like klass is an invalid pointer in the following function: void MetaspaceShared::link_shared_classes(TRAPS) { // Collect all loaded ClassLoaderData. ResourceMark rm; LambdaFormInvokers::regenerate_holder_classes(CHECK); CollectCLDClosure collect_cld; { // ClassLoaderDataGraph::loaded_cld_do requires ClassLoaderDataGraph_lock. // We cannot link the classes while holding this lock (or else we may run into deadlock). // Therefore, we need to first collect all the CLDs, and then link their classes after // releasing the lock. MutexLocker lock(ClassLoaderDataGraph_lock); ClassLoaderDataGraph::loaded_cld_do(&collect_cld); } while (true) { bool has_linked = false; for (int i = 0; i < collect_cld.nof_cld(); i++) { ClassLoaderData* cld = collect_cld.cld_at(i); for (Klass* klass = cld->klasses(); klass != NULL; klass = klass->next_link()) { if (klass->is_instance_klass()) { <<<< crashes here InstanceKlass* ik = InstanceKlass::cast(klass); if (may_be_eagerly_linked(ik)) { has_linked |= link_class_for_cds(ik, CHECK); } } } } if (!has_linked) { break; } // Class linking includes verification which may load more classes. // Keep scanning until we have linked no more classes. } }
09-09-2021