Bug ID: JDK-8233491 Crash in AdapterHandlerLibrary::get_adapter with CDS due to code cache exhaustion

JDK-8233491 : Crash in AdapterHandlerLibrary::get_adapter with CDS due to code cache exhaustion

Type: Bug
Component: hotspot
Sub-Component: compiler
Affected Version: 11,12,13,14

Priority: P4
Status: Resolved
Resolution: Fixed

Submitted: 2019-11-04
Updated: 2022-06-27
Resolved: 2019-11-07

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 11	JDK 13	JDK 14
11.0.7-oracleFixed	13.0.4Fixed	14 b23Fixed

Related Reports

Relates :

JDK-8058176 - [mlvm] tests should not allow code cache exhaustion

Description

JDK 14 CI test failure:

vmTestbase/vm/mlvm/meth/stress/compiler/deoptimize/Test.java#id1

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffa18130c97, pid=137924, tid=142928
#
# JRE version: Java(TM) SE Runtime Environment (14.0+22) (fastdebug build 14-ea+22-940)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 14-ea+22-940, compiled mode, sharing, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# V  [jvm.dll+0xb90c97]  AdapterHandlerLibrary::get_adapter+0x207
#
# Core dump will be written. Default location: T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\scratch\1\hs_err_pid137924.mdmp
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Dtest.class.path.prefix=T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\classes\5\vmTestbase\vm\mlvm\meth\stress\compiler\deoptimize\Test_id1.d;C:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\src.full\open\test\hotspot\jtreg\vmTestbase\vm\mlvm\meth\stress\compiler\deoptimize;T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\classes\5\vmTestbase;T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\classes\5\test\lib -Dtest.src=C:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\src.full\open\test\hotspot\jtreg\vmTestbase\vm\mlvm\meth\stress\compiler\deoptimize -Dtest.src.path=C:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\src.full\open\test\hotspot\jtreg\vmTestbase\vm\mlvm\meth\stress\compiler\deoptimize;C:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\src.full\open\test\hotspot\jtreg\vmTestbase;C:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\src.full\open\test\lib -Dtest.classes=T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\classes\5\vmTestbase\vm\mlvm\meth\stress\compiler\deoptimize\Test_id1.d -Dtest.class.path=T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\classes\5\vmTestbase\vm\mlvm\meth\stress\compiler\deoptimize\Test_id1.d;T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\classes\5\vmTestbase;T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\classes\5\test\lib -Dtest.vm.opts=-XX:MaxRAMPercentage=3 -Dtest.tool.vm.opts=-J-XX:MaxRAMPercentage=3 -Dtest.compiler.opts= -Dtest.java.opts=-Xcomp -XX:+CreateCoredumpOnCrash -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:+TieredCompilation -XX:+DeoptimizeALot -Dtest.jdk=c:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\windows-x64-debug.jdk\jdk-14\fastdebug -Dcompile.jdk=c:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\windows-x64-debug.jdk\jdk-14\fastdebug -Dtest.timeout.factor=10.0 -Dtest.root=C:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\src.full\open\test\hotspot\jtreg -Dtest.nativepath=c:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\windows-x64-debug.test\hotspot\jtreg\native -XX:MaxRAMPercentage=3 -Xcomp -XX:+CreateCoredumpOnCrash -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -XX:+TieredCompilation -XX:+DeoptimizeALot -Djava.library.path=c:\ade\mesos\work_dir\jib-master\install\jdk-14+22-940\windows-x64-debug.test\hotspot\jtreg\native -XX:ReservedCodeCacheSize=100m com.sun.javatest.regtest.agent.MainWrapper T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\vmTestbase\vm\mlvm\meth\stress\compiler\deoptimize\Test_id1.d\main.0.jta -threadsPerCpu 2 -threadsExtra 2

Host: inst-7iepi-Win2, AMD EPYC 7551 32-Core Processor                , 16 cores, 63G,  Windows Server 2012 R2 , 64 bit Build 9600 (6.3.9600.19358)
Time: Sat Nov  2 03:11:18 2019 /GM elapsed time: 271 seconds (0d 0h 4m 31s)

---------------  T H R E A D  ---------------

Current thread (0x0000004d4bf6d000):  JavaThread "Thread-18" [_thread_in_vm, id=142928, stack(0x0000004d50080000,0x0000004d50180000)]

Stack: [0x0000004d50080000,0x0000004d50180000],  sp=0x0000004d5017de30,  free space=1015k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0xb90c97]  AdapterHandlerLibrary::get_adapter+0x207  (sharedruntime.cpp:2628)
V  [jvm.dll+0xa5abe1]  Method::make_adapters+0x21  (method.cpp:1211)
V  [jvm.dll+0xa5a48b]  Method::link_method+0x27b  (method.cpp:1205)
V  [jvm.dll+0xa5e0a4]  Method::restore_unshareable_info+0xd4  (method.cpp:1240)
V  [jvm.dll+0x6b9295]  InstanceKlass::restore_unshareable_info+0x105  (instanceklass.cpp:2398)
V  [jvm.dll+0xc37b26]  SystemDictionary::load_shared_class+0x326  (systemdictionary.cpp:1336)
V  [jvm.dll+0xc3752e]  SystemDictionary::load_instance_class+0x1ae  (systemdictionary.cpp:1448)
V  [jvm.dll+0xc3970f]  SystemDictionary::resolve_instance_class_or_null+0x89f  (systemdictionary.cpp:848)
V  [jvm.dll+0xc39bcc]  SystemDictionary::resolve_or_fail+0x4c  (systemdictionary.cpp:196)
V  [jvm.dll+0x4f5479]  ConstantPool::klass_at_impl+0x3a9  (constantpool.cpp:498)
V  [jvm.dll+0x6c826a]  InterpreterRuntime::_new+0xca  (interpreterruntime.cpp:230)
C  0x0000004d3b32c32c

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  java.util.Formatter.parse(Ljava/lang/String;)Ljava/util/List;+81 java.base@14-ea
j  java.util.Formatter.format(Ljava/util/Locale;Ljava/lang/String;[Ljava/lang/Object;)Ljava/util/Formatter;+12 java.base@14-ea
j  java.util.Formatter.format(Ljava/lang/String;[Ljava/lang/Object;)Ljava/util/Formatter;+7 java.base@14-ea
j  java.lang.String.format(Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String;+9 java.base@14-ea
j  nsk.share.test.LazyFormatString.toString()Ljava/lang/String;+8
J 23449 c2 java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; java.base@14-ea (15 bytes) @ 0x0000004d3d075e38 [0x0000004d3d075a20+0x0000000000000418]
j  java.io.PrintWriter.println(Ljava/lang/Object;)V+1 java.base@14-ea
j  nsk.share.Log.printExceptionToString(Ljava/lang/Object;Ljava/lang/Throwable;)Ljava/lang/String;+19
j  nsk.share.Log.complain(Ljava/lang/Object;Ljava/lang/Throwable;)V+7
j  vm.mlvm.share.Env.complain(Ljava/lang/Throwable;Ljava/lang/String;[Ljava/lang/Object;)V+13
j  vm.mlvm.share.MultiThreadedTest.lambda$run$1(Ljava/util/concurrent/CyclicBarrier;I)V+54
J 4212 c1 vm.mlvm.share.MultiThreadedTest$$Lambda$6.run()V (16 bytes) @ 0x0000004d3be01404 [0x0000004d3be01300+0x0000000000000104]
J 2146 c1 java.lang.Thread.run()V java.base@14-ea (17 bytes) @ 0x0000004d3bbffaac [0x0000004d3bbff9e0+0x00000000000000cc]
v  ~StubRoutines::call_stub

siginfo: EXCEPTION_ACCESS_VIOLATION (0xc0000005), reading address 0x0000000000000028
...............

Comments

Fix Request (13u) This fixes the corner case that crashes the VM. Patch applies cleanly to 13u, passes tier1
03-06-2020
Fix Request (11u) This fixes the corner case that crashes the VM, and keeps codebases in sync (I see 11.0.7-oracle). Patch applies cleanly to 11u, passes tier1 and tier2 tests.
15-01-2020
URL: https://hg.openjdk.java.net/jdk/jdk/rev/00244fd3169a User: thartmann Date: 2019-11-07 06:03:25 +0000
07-11-2019
When running a stress test with CDS, we fail to create adapters when linking a method from a shared class because the code cache is full. This case is not properly handled by the CDS specific code and instead of throwing a VirtualMachineError, we crash because "entry" is NULL. http://cr.openjdk.java.net/~thartmann/8233491/webrev.00/
06-11-2019
I can reproduce the crash with attached Test.java: java -Xshare:on -XX:ReservedCodeCacheSize=2000k -XX:InitialCodeCacheSize=2000k -XX:-ClassUnloading Test 1 [0,473s][warning][codecache] CodeCache is full. Compiler has been disabled. [0,473s][warning][codecache] Try increasing the code cache size using -XX:ReservedCodeCacheSize= Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= CodeCache: size=2000Kb used=1985Kb max_used=1996Kb free=14Kb bounds [0x00007f411a7c2000, 0x00007f411a9b6000, 0x00007f411a9b6000] total_blobs=1347 nmethods=106 adapters=881 compilation: disabled (not enough contiguous free space left) stopped_count=1, restarted_count=0 full_count=0 # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f4129bd0ead, pid=32524, tid=32525 # # JRE version: Java(TM) SE Runtime Environment (14.0) (fastdebug build 14-internal+0-adhoc.tobias.open) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 14-internal+0-adhoc.tobias.open, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64) # Problematic frame: # V [libjvm.so+0x15caead] AdapterHandlerLibrary::get_adapter(methodHandle const&)+0x51d # # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /oracle/jdk_jdk/open/core.32524) # # An error report file with more information is saved as: # /oracle/jdk_jdk/open/hs_err_pid32524.log # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # Current thread is 32525 Dumping core ... Aborted (core dumped) But the test depends on the class loading sequence and is too unstable to be converted to a jtreg test (I've tried for a long time).
06-11-2019
I don't think this is a duplicate of JDK-8058176. The VM should not crash but throw an error if we run out of code cache space during adapter generation. Of course, this VirtualMachineError then needs to be handled by the tests.
06-11-2019
maybe duplicate of JDK-8058176 !?
06-11-2019
It seems like we have run out of code cache: CodeCache: size=102400Kb used=102400Kb max_used=102400Kb free=0Kb bounds [0x0000004d3b300000, 0x0000004d41700000, 0x0000004d41700000] total_blobs=58443 nmethods=17552 adapters=9044 compilation: disabled (not enough contiguous free space left) stopped_count=1, restarted_count=0 full_count=17 This caused AdapterHandlerLibrary::get_adapter0 to return NULL if (new_adapter == NULL) { // CodeCache is full, disable compilation // Ought to log this but compile log is only per compile thread // and we're some non descript Java thread. return NULL; // Out of CodeCache space } Reassigned to compiler team for further evaluation.
05-11-2019
Host: XXXXXXX, AMD EPYC 7551 32-Core Processor, 16 cores, 63G, Windows Server 2012 R2 , 64 bit Build 9600 (6.3.9600.19358) --------------- T H R E A D --------------- Current thread (0x0000004d4bf6d000): JavaThread "Thread-18" [_thread_in_vm, id=142928, stack(0x0000004d50080000,0x0000004d50180000)] Stack: [0x0000004d50080000,0x0000004d50180000], sp=0x0000004d5017de30, free space=1015k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xb90c97] AdapterHandlerLibrary::get_adapter+0x207 (sharedruntime.cpp:2628) V [jvm.dll+0xa5abe1] Method::make_adapters+0x21 (method.cpp:1211) V [jvm.dll+0xa5a48b] Method::link_method+0x27b (method.cpp:1205) V [jvm.dll+0xa5e0a4] Method::restore_unshareable_info+0xd4 (method.cpp:1240) V [jvm.dll+0x6b9295] InstanceKlass::restore_unshareable_info+0x105 (instanceklass.cpp:2398) V [jvm.dll+0xc37b26] SystemDictionary::load_shared_class+0x326 (systemdictionary.cpp:1336) V [jvm.dll+0xc3752e] SystemDictionary::load_instance_class+0x1ae (systemdictionary.cpp:1448) V [jvm.dll+0xc3970f] SystemDictionary::resolve_instance_class_or_null+0x89f (systemdictionary.cpp:848) V [jvm.dll+0xc39bcc] SystemDictionary::resolve_or_fail+0x4c (systemdictionary.cpp:196) V [jvm.dll+0x4f5479] ConstantPool::klass_at_impl+0x3a9 (constantpool.cpp:498) V [jvm.dll+0x6c826a] InterpreterRuntime::_new+0xca (interpreterruntime.cpp:230) C 0x0000004d3b32c32c Looks like the variable 'entry' is NULL at this line: http://hg.openjdk.java.net/jdk/jdk/file/2938e0a4e954/src/hotspot/share/runtime/sharedRuntime.cpp#l2628 2628: SharedRuntime::generate_trampoline(&_masm, entry->get_c2i_entry());
05-11-2019
ILW = HLM = P3
05-11-2019