JDK-8224692 : runtime/appcds tests crash in "HotSpotJVMCI::compute_offset" when running in Graal as JIT mode
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 13
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2019-05-23
  • Updated: 2020-03-09
  • Resolved: 2019-05-31
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 13 JDK 14
13 b24Fixed 14Fixed
Related Reports
Relates :  
JDK-8240548 - [TESTBUG] runtime/cds/appcds/dynamicArchive/NoClassToArchive.java fails with Graal
Relates :  
JDK-8207812 - Implement Dynamic CDS Archive
Description
A lot of runtime/appcds tests started to crash in latest jdk13 bits. No issue is present in latest promoted jdk13b22 build.

The crash is
#  Internal Error (open/src/hotspot/share/jvmci/jvmciJavaClasses.cpp:70), pid=12270, tid=12284
#  fatal error: symbol with name metadataHandle and signature J was not found in symbol table (klass=jdk/vm/ci/hotspot/HotSpotConstantPool)
#
# JRE version: Java(TM) SE Runtime Environment (13.0) (fastdebug build 13-internal+0-2019-05-23-1918462.katya.null)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 13-internal+0-2019-05-23-1918462.katya.null, mixed mode, sharing, tiered, jvmci, jvmci compiler, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xffdeec]  HotSpotJVMCI::compute_offset(int&, Klass*, char const*, char const*, bool, Thread*)+0x2dc
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /home/katya/work/JavaSE/Hotspot/ws/jdk.jdk/JTwork/scratch/0/core.12270)
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -ea -esa -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI -XX:+TieredCompilation -XX:+UseJVMCICompiler -Djvmci.Compiler=graal -XX:ArchiveClassesAtExit=/home/katya/work/JavaSE/Hotspot/ws/jdk.jdk/JTwork/scratch/0/appcds-12h36m23s577.jsa -Xshare:auto -Xlog:cds+dynamic=debug ArrayKlassesApp

Host: olegbox, Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz, 24 cores, 15G, Ubuntu 16.04.3 LTS
Time: Thu May 23 12:36:24 2019 PDT elapsed time: 0 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x00007f120c298800):  JavaThread "JVMCI CompilerThread0" daemon [_thread_in_vm, id=12284, stack(0x00007f11d4b9e000,0x00007f11d4c9f000)]

Current CompileTask:
JVMCI:    579  188       4       java.lang.String::charAt (25 bytes)

Stack: [0x00007f11d4b9e000,0x00007f11d4c9f000],  sp=0x00007f11d4c9d550,  free space=1021k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xffdeec]  HotSpotJVMCI::compute_offset(int&, Klass*, char const*, char const*, bool, Thread*)+0x2dc
V  [libjvm.so+0x10014c4]  HotSpotJVMCI::compute_offsets(Thread*)+0x33b4
V  [libjvm.so+0x1051a54]  JVMCIRuntime::initialize(JVMCIEnv*)+0x274
V  [libjvm.so+0x1054f2a]  JVMCIRuntime::compile_method(JVMCIEnv*, JVMCICompiler*, methodHandle const&, int)+0xfa
V  [libjvm.so+0x9cec3b]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0xd4b
V  [libjvm.so+0x9cf088]  CompileBroker::compiler_thread_loop()+0x358
V  [libjvm.so+0x16bf366]  JavaThread::thread_main_inner()+0x226
V  [libjvm.so+0x16c49a6]  Thread::call_run()+0xf6
V  [libjvm.so+0x13df8be]  thread_native_entry(Thread*)+0x10e
Comments
http://cr.openjdk.java.net/~iklam/jdk13/8224692-dynamic-cds-tests-crash-with-graal-jit.v01/ The dynamic archive is created inside the before_exit() function. This is a convenient place to do it -- the JVM could exit in multiple ways, but they all come to this function. The down side of the before_exit() function is that some small amount Java code could run afterwards, including code executed by the Graal JIT. The design of Dynamic CDS *should* support this -- the archive creation happens inside a safepoint, saving a copy of the loaded classes into the archive file. It should not have any side effects observable from the Java code. As a result, it should be possible to resume the execution of Java code after the archive is created. The bug here is we incorrectly reset the shared symbol table during archive creation. The fix is pretty simple -- don't do that. I also fixed two other bugs that surface only with Graal JIT: + Avoid having an empty "MC" region. + In the test NoClassToArchive.java, when Graal JIT is used, anonymous classes might be loaded.
30-05-2019
Okay so it occurs at its own safepoint - all good then.
28-05-2019
Dynamic dumping runs inside a VM_Operation -- it walks over all loaded classes (in theory, without modifying anything), and writes a copy of these classes into a file. So it should be as safe as other VM_Ops such as VM_PrintClassHierarchy::doit(). The only impact to any Java code (including Graal JIT) would be a VM_Op that potentially may take a long time to complete. I suspect the bug is that, contrary to the design, we inadvertently modified some states that are observable by Graal JIT.
28-05-2019
So is dumping the dynamic archive "locking out" classloading (or some related activity) so it can do the dump, which then interferes with other threads that are still continuing execution in parallel with the dump? This would impact not only JIT threads but also application threads in general. I would have expected a "dump at exit" to occur when we have reached the termination safepoint so such interference is precluded.
28-05-2019
The HelloDynamic.java test case shows the problem more clearly. From the jtr file: 1.296s][info ][cds ] Shared file region 0: 0x00009e10 bytes, addr 0x0000000800c3e000 file offset 0x000f7000 [1.296s][info ][cds,dynamic] Written dynamic archive 0x0000000800b48000 - 0x0000000800c47e10 [992 bytes header, 1048080 bytes total] [1.296s][info ][cds,dynamic] 143 klasses; 3171 symbols [1.296s][info ][cds ] Verify After CDS dynamic dump java.lang.NoClassDefFoundError: java/lang/Object java.lang.NoClassDefFoundError: java/lang/Object at jdk.vm.ci.hotspot.amd64.AMD64HotSpotJVMCIBackendFactory.createTarget(jdk.internal.vm.ci@13-internal/AMD64HotSpotJVMCIBackendFactory.java:148) at jdk.vm.ci.hotspot.amd64.AMD64HotSpotJVMCIBackendFactory.createJVMCIBackend(jdk.internal.vm.ci@13-internal/AMD64HotSpotJVMCIBackendFactory.java:183) at jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.<init>(jdk.internal.vm.ci@13-internal/HotSpotJVMCIRuntime.java:458) at jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.runtime(jdk.internal.vm.ci@13-internal/HotSpotJVMCIRuntime.java:178) at jdk.vm.ci.runtime.JVMCI.initializeRuntime(jdk.internal.vm.ci@13-internal/Native Method) at jdk.vm.ci.runtime.JVMCI.getRuntime(jdk.internal.vm.ci@13-internal/JVMCI.java:65) # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/jvmciRuntime.cpp:910 # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (open/src/hotspot/share/jvmci/jvmciRuntime.cpp:910), pid=22523, tid=22546 # guarantee(false) failed: Failed registering CompilerToVM native methods # # JRE version: Java(TM) SE Runtime Environment (13.0) (fastdebug build 13-internal+0-2019-05-23-1735120.ekaterina.pavlova.jdk.jdk) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 13-internal+0-2019-05-23-1735120.ekaterina.pavlova.jdk.jdk, mixed mode, sharing, tiered, jvmci, jvmci compiler, compressed oops, g1 gc, linux-amd64) # Problematic frame: # V [libjvm.so+0x1055e19] JVM_RegisterJVMCINatives+0x4c9 #
27-05-2019
The crash happened after the Dynamic CDS archive has been dumped (with -XX:ArchiveClassesAtExit). This happens here in before_exit(): http://hg.openjdk.java.net/jdk/jdk/file/8b43726c1a47/src/hotspot/share/runtime/java.cpp#l504 Apparently the graal compiler continues to execute after the Dynamic CDS archive has been dumped. This *should* be OK as the design of the dynamic CDS dumping process is not supposed to modify anything observable from Java, so Graal, or any existing Java programs, should be able to continue execution. However, that doesn't seem to be the case. Excepts from hs_err_bug8224692.log ---- # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 13-internal+0-2019-05-23-1918462.katya.null, mixed mode, sharing, tiered, jvmci, jvmci compiler, compressed oops, g1 gc, linux-amd64) Command Line: -ea -esa -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI -XX:+TieredCompilation -XX:+UseJVMCICompiler -Djvmci.Compiler=graal -XX:ArchiveClassesAtExit=/home/katya/work/JavaSE/Hotspot/ws/jdk.jdk/JTwork/scratch/0/appcds-12h36m23s577.jsa -Xshare:auto -Xlog:cds+dynamic=debug ArrayKlassesApp 7ffe00000-7ffe48000 rw-p 00bce000 08:05 27946538 /home/katya/work/JavaSE/Hotspot/ws/jdk.jdk/build/linux-x64/images/jdk/lib/server/classes.jsa ------------- Excerpts from the corresponding jtr file: ----- [1.336s][debug ][cds,dynamic] klasses[ 95] = 0x0000000800b77038 java.util.AbstractList$SubList$1 [1.336s][debug ][cds,dynamic] klasses[ 96] = 0x0000000800b784d8 java.lang.invoke.DirectMethodHandle$Interface [1.336s][debug ][cds,dynamic] klasses[ 97] = 0x0000000800b78df0 java.lang.module.Resolver [1.346s][info ][cds,dynamic] Written dynamic archive 0x0000000800b48000 - 0x0000000800be98b8 [992 bytes header, 661688 bytes total] [1.346s][info ][cds,dynamic] 98 klasses; 1784 symbols jdk.vm.ci.hotspot.HotSpotMetaspaceConstantImpl {0x0000000800b5f780} - instance size: 3 - klass size: 84 - access: final synchronized - state: fully_initialized - name: 'jdk/vm/ci/hotspot/HotSpotMetaspaceConstantImpl' - super: 'java/lang/Object' - sub: - arrays: NULL - methods: Array<T>(0x00007f8abb937b58) - method ordering: Array<T>(0x0000000800ab65d8) - default_methods: Array<T>(0x0000000000000000) - local interfaces: Array<T>(0x00007f8abb937868) - trans. interfaces: Array<T>(0x00007f8abb938770) - constants: constant pool [107] {0x00007f8abb937460} for 'jdk/vm/ci/hotspot/HotSpotMetaspaceConstantImpl' cache=0x00007f8abb9387c0 - class loader data: loader data: 0x00007f8af429a600 of 'bootstrap' - unsafe anonymous host class: NULL - source file: 'HotSpotMetaspaceConstantImpl.java' - class annotations: Array<T>(0x0000000000000000) - class type annotations: Array<T>(0x0000000000000000) - field annotations: Array<T>(0x0000000000000000) - field type annotations: Array<T>(0x0000000000000000) - inner classes: Array<T>(0x0000000800592688) - nest members: Array<T>(0x0000000800592688) - java mirror: a 'java/lang/Class'{0x00000007866efd78} = 'jdk/vm/ci/hotspot/HotSpotMetaspaceConstantImpl' - vtable length 5 (start addr: 0x0000000800b5f960) - itable length 17 (start addr: 0x0000000800b5f988) - ---- static fields (1 words): - static final synthetic '$assertionsDisabled' 'Z' @112 - ---- non-static fields (2 words): - private final 'compressed' 'Z' @12 - private final 'metaspaceObject' 'Ljdk/vm/ci/hotspot/MetaspaceObject;' @16 - non-static oop maps: 16-16 # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/jvmciJavaClasses.cpp:70 # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (open/src/hotspot/share/jvmci/jvmciJavaClasses.cpp:70), pid=22208, tid=22230 # fatal error: symbol with name compressed and signature Z was not found in symbol table (klass=jdk/vm/ci/hotspot/HotSpotMetaspaceConstantImpl) -----
27-05-2019
These are new tests that were added on 5/17 and started failing in the Graal nightly on 5/18. I see no regression, just a lack of testing with Graal before pushing the new tests. I suggest we problem-list these tests for Graal until we have a fix.
27-05-2019
[~iklam][~jiangli][~ccheung] Is Dynamic CDS stripping symbols that Graal/JVMCI might need?
27-05-2019
Sorry, I don't have experience with AppCDS which seems to be stripping away some symbols (if I interpret the crash properly).
24-05-2019
To reproduce run for example: > make run-test TEST_VM_OPTS="-server -ea -esa -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI -XX:+TieredCompilation -XX:+UseJVMCICompiler -Djvmci.Compiler=graal" TEST="runtime/appcds/dynamicArchive/ArrayKlasses.java"
23-05-2019