JDK-8274338 : com/sun/jdi/RedefineCrossEvent.java failed "assert(m != __null) failed: NULL mirror"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 18
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2021-09-27
  • Updated: 2022-01-13
  • Resolved: 2021-10-15
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 17 JDK 18
11.0.14Fixed 17.0.2Fixed 18 b20Fixed
Related Reports
Relates :  
JDK-8275318 - loaded_classes_do may see ArrayKlass before InstanceKlass is loaded
Relates :  
JDK-8274753 - ZGC: SEGV in MetaspaceShared::link_shared_classes
Relates :  
JDK-8276943 - Backout JDK-8274338 11u backport
Description
The following test failed in the JDK18 CI:

com/sun/jdi/RedefineCrossEvent.java

Here's a snippet from the log file:

----------System.out:(30/2488)----------
vmOpts: '-Xmx768m -XX:MaxRAMPercentage=4.16667 -Djava.io.tmpdir=/opt/mach5/mesos/work_dir/slaves/ff806ead-2cac-495d-9cbc-62116f99bf14-S13934/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/e0a31b42-087a-47ac-bcb3-5dfbecc2c39b/runs/a4f5fde5-fe07-4985-8e04-96641cfc5aed/testoutput/test-support/jtreg_open_test_jdk_jdk_jdi/tmp -ea -esa'
javaOpts: '-XX:+CreateCoredumpOnCrash -XX:+UseZGC'
JVM version:18-ea
JDI version: 18.0
JVM description: Java Debug Interface (Reference Implementation) version 18.0 
Java Debug Wire Protocol (Reference Implementation) version 18.0
JVM Debug Interface version 18.0
JVM version 18-ea (Java HotSpot(TM) 64-Bit Server VM, mixed mode, sharing)
Howdy!
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/classLoaderData.cpp:362
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/opt/mach5/mesos/work_dir/slaves/a2dc162d-743b-4800-9e92-31f85abb45b1-S1317/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6cc982f3-0197-4d8f-b494-7dc978f0fec0/runs/e0a416ab-5d4f-4994-aa46-3742148ba1d7/workspace/open/src/hotspot/share/classfile/classLoaderData.cpp:362), pid=3968, tid=4018
#  assert(m != __null) failed: NULL mirror
#
# JRE version: Java(TM) SE Runtime Environment (18.0+17) (fastdebug build 18-ea+17-971)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 18-ea+17-971, mixed mode, sharing, tiered, compressed class ptrs, z gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x97c055]  ClassLoaderData::loaded_classes_do(KlassClosure*)+0x245
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/ff806ead-2cac-495d-9cbc-62116f99bf14-S13934/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/e0a31b42-087a-47ac-bcb3-5dfbecc2c39b/runs/a4f5fde5-fe07-4985-8e04-96641cfc5aed/testoutput/test-support/jtreg_open_test_jdk_jdk_jdi/scratch/0/core.3968)
#
# An error report file with more information is saved as:
# /opt/mach5/mesos/work_dir/slaves/ff806ead-2cac-495d-9cbc-62116f99bf14-S13934/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/e0a31b42-087a-47ac-bcb3-5dfbecc2c39b/runs/a4f5fde5-fe07-4985-8e04-96641cfc5aed/testoutput/test-support/jtreg_open_test_jdk_jdk_jdi/scratch/0/hs_err_pid3968.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
----------System.err:(24/1767)----------
[0ms] run args: [-redefstart, -redefevent, ClassesByName2Targ]
[316ms] Redefining class ClassesByName2Targ (loaded by instance of jdk.internal.loader.ClassLoaders$AppClassLoader(id=47))
[319ms] Redefining class ClassesByName2Targ (loaded by instance of jdk.internal.loader.ClassLoaders$AppClassLoader(id=47))
com.sun.jdi.VMDisconnectedException
	at jdk.jdi/com.sun.tools.jdi.TargetVM.waitForReply(TargetVM.java:310)
	at jdk.jdi/com.sun.tools.jdi.VirtualMachineImpl.waitForTargetReply(VirtualMachineImpl.java:1173)
	at jdk.jdi/com.sun.tools.jdi.PacketStream.waitForReply(PacketStream.java:89)
	at jdk.jdi/com.sun.tools.jdi.JDWP$VirtualMachine$AllClassesWithGeneric.waitForReply(JDWP.java:1610)
	at jdk.jdi/com.sun.tools.jdi.JDWP$VirtualMachine$AllClassesWithGeneric.process(JDWP.java:1596)
	at jdk.jdi/com.sun.tools.jdi.VirtualMachineImpl.retrieveAllClasses(VirtualMachineImpl.java:1143)
	at jdk.jdi/com.sun.tools.jdi.VirtualMachineImpl.allClasses(VirtualMachineImpl.java:338)
	at ClassesByName2Test.runTests(ClassesByName2Test.java:163)
	at TestScaffold.startTests(TestScaffold.java:431)
	at ClassesByName2Test.main(ClassesByName2Test.java:123)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
	at java.base/java.lang.Thread.run(Thread.java:833)

JavaTest Message: Test threw exception: com.sun.jdi.VMDisconnectedException
JavaTest Message: shutting down test

result: Failed. Execution failed: `main' threw exception: com.sun.jdi.VMDisconnectedException


Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x00007f2bbc2b1e70):  JavaThread "JDWP Transport Listener: dt_socket" daemon [_thread_in_vm, id=4018, stack(0x00007f2ba43f8000,0x00007f2ba44f9000)]

Stack: [0x00007f2ba43f8000,0x00007f2ba44f9000],  sp=0x00007f2ba44f7860,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x97c055]  ClassLoaderData::loaded_classes_do(KlassClosure*)+0x245
V  [libjvm.so+0x986c2d]  ClassLoaderDataGraph::loaded_classes_do(KlassClosure*)+0x1bd
V  [libjvm.so+0x11e0258]  JvmtiGetLoadedClasses::getLoadedClasses(JvmtiEnv*, int*, _jclass***)+0xd8
V  [libjvm.so+0x114077a]  jvmti_GetLoadedClasses+0x10a
C  [libjdwp.so+0xce19]  allClasses1.part.0+0x39
C  [libjdwp.so+0xde73]  allClassesWithGeneric+0x23
C  [libjdwp.so+0x134a9]  debugLoop_run+0x299
C  [libjdwp.so+0x26ef4]  attachThread+0x54
V  [libjvm.so+0x11e1c5b]  JvmtiAgentThread::call_start_function()+0x17b
V  [libjvm.so+0x18910ec]  JavaThread::thread_main_inner()+0x27c
V  [libjvm.so+0x1897690]  Thread::call_run()+0x100
V  [libjvm.so+0x15753d4]  thread_native_entry(Thread*)+0x104

Since this assertion failure happened in
ClassLoaderData::loaded_classes_do(), I'm starting this
bug off in hotspot/runtime for initial triage.
Comments
Zhengyu, thanks for addressing this again!
13-01-2022
Fix Request (11u Redo) I would like to backport this patch to openjdk 11u for parity with Oracle 11.0.15. The patch fixed a subtle race that can result fatal error. Early JDK11u backport failed and had been backed out by JDK-8276943 . This redo-patch has been reviewed by @phh.
10-01-2022
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk11u-dev/pull/632 Date: 2021-11-16 15:17:24 +0000
07-01-2022
Fix Request (11u) I would like to backport this patch to openjdk 11u for parity with Oracle 11.0.15. The patch fixed a subtle race that can result fatal error. The original patch applies cleanly, but needs to take additional MultiArray_lock on CDS path to get it work correctly. The 11u patch has been reviewed by @phh
05-11-2021
Fix Request (17u) A clean backport for parity with Oracle 17.0.3. The patch fixed a subtle race that can result fatal error.
03-11-2021
Changeset: 172aed1a Author: Coleen Phillimore <coleenp@openjdk.org> Date: 2021-10-15 12:10:27 +0000 URL: https://git.openjdk.java.net/jdk/commit/172aed1a2d75756b140cb723133ac5fb67f7745e
15-10-2021
105 // To get a consistent list of classes we need MultiArray_lock to ensure 106 // array classes aren't created. 107 MutexLocker ma(MultiArray_lock); 108 InstanceKlass::restore_unshareable_info doesn't lock MultiArray_lock before restoring array classes. The lock is held when creating array classes. The class that fails is shared: (gdb) print /x k->_access_flags._flags $3 = 0x82000000 (gdb) print (char*)k->_name->_body $4 = 0x8004a21ee "[Ljava/lang/invoke/LambdaForm$NamedFunction;" Since this is a crash that happened essentially once, this could be the reason for this rare race.
13-10-2021
One thread is doing JVMTI GetLoadedClasses, so walks the ClassLoaderDataGraph with loaded_classes_do: This is the assert it gets in ClassLoaderData::loaded_classes_do, where this is the NULL CLD. 359 if (k->is_array_klass() || (k->is_instance_klass() && InstanceKlass::cast(k)->is_loaded())) { 360 #ifdef ASSERT 361 oop m = k->java_mirror(); 362 assert(m != NULL, "NULL mirror"); The class is an ObjArrayKlass, and in gdb the mirror looks perfectly good: (gdb) print k->_java_mirror->_obj._o $350 = (oopDesc *) 0x100000805940 The other mirrors in the previous classes that it walked look similar: (gdb) set $k = _klasses (gdb) while ($k != k) > print $k->_java_mirror->_obj._o > set $k = $k->_next_link >end $293 = (oopDesc *) 0x100000a0f900 $294 = (oopDesc *) 0x100000c04118 $295 = (oopDesc *) 0x100000c03fe0 ... $348 = (oopDesc *) 0x100000a016b0 $349 = (oopDesc *) 0x100000c00b50 The only thing that looks suspicious is that in InstanceKlass, we link the klass into the ClassLoaderData::_klasses after we create the mirror, and for ObjArrayKlass, we link them before we create the mirror. In the former case, we test is_loaded() but in the ObjArrayKlass we cannot. So maybe there's a race here. The other thread in this core file is loading classes from the shared archive, which has the same timing, but not this class.
13-10-2021
According to the hs_err file this crashes very early, before any GC has run: GC Heap History (0 events): No events also: Classes unloaded (0 events): No events I don't think this is related to class unloading.
06-10-2021
class ClassLoaderDataGraphIterator : public StackObj { ... // Skip already unloaded CLD for concurrent unloading. while (cld != NULL && !cld->is_alive()) { cld = cld->next(); } if (cld != NULL) { // Keep cld that is being returned alive. _holder = Handle(_thread, cld->holder_phantom()); _next = cld->next(); This should keep the cld alive while iterating, so why is it unloaded here?
05-10-2021
ILW = HLM = P3
28-09-2021