JDK-8295097 : DirectNativeCallWrapper::verify_resolve_call fails with "assert(db != __null && !db->is_adapter_blob()) failed: must use stub!"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 11.0.18-oracle,20
  • Priority: P2
  • Status: Closed
  • Resolution: Duplicate
  • OS: linux
  • CPU: aarch64
  • Submitted: 2022-10-10
  • Updated: 2022-12-20
  • Resolved: 2022-12-20
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 20
20Resolved
Related Reports
Blocks :  
Duplicate :  
Duplicate :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Description
The following test failed in the JDK20 CI:

runtime/cds/appcds/redefineClass/RedefineRunningMethods_Shared.java

Here's a snippet from the log file:

----------System.err:(155/12572)----------
 stdout: [The class is shared as expected: RedefineRunningMethods_Shared
The class is shared as expected: RedefineRunningMethods_SharedHelper
The class is shared as expected: RedefineRunningMethods
The class is shared as expected: RedefineRunningMethods_B
The class is shared as expected: RedefineClassHelper
The class is shared as expected: jdk.test.lib.compiler.InMemoryJavaCompiler
The class is shared as expected: jdk.test.lib.compiler.InMemoryJavaCompiler$FileManagerWrapper
The class is shared as expected: jdk.test.lib.compiler.InMemoryJavaCompiler$FileManagerWrapper$1
The class is shared as expected: jdk.test.lib.compiler.InMemoryJavaCompiler$MemoryJavaFileObject
[2.624s][trace][redefine,class,iklass,add] adding previous version ref for RedefineRunningMethods_B, EMCP_cnt=4
[2.624s][trace][redefine,class,iklass,add] scratch class added; one of its methods is on_stack.
[2.654s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.654s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.654s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.654s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
infinite called
[2.693s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.693s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.693s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.693s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.715s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.715s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.715s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.715s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.740s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.740s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.740s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.740s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.762s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.762s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.762s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.762s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.787s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.787s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.787s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.787s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.811s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.811s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.811s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.811s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.834s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.834s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.834s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.834s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.856s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.856s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.856s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.856s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.878s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.878s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.878s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.879s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.902s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.902s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.902s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.902s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.943s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.943s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.943s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.943s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.966s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.966s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.966s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.966s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[2.988s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[2.988s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[2.988s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[2.988s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.024s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.024s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.024s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.024s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.048s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.048s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.048s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.048s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.073s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.073s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.073s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.073s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.097s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.097s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.097s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.097s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.121s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.121s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.121s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.121s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.144s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.144s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.144s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.144s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.172s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.172s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.172s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.172s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.295s][trace][redefine,class,iklass,add  ] adding previous version ref for RedefineRunningMethods_B, EMCP_cnt=2
[3.295s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.295s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.295s][trace][redefine,class,iklass,purge] previous version stats: live=1, deleted=0
[3.295s][trace][redefine,class,iklass,add  ] infinite_emcp(()V): flush obsolete method @3 in version @0
[3.295s][trace][redefine,class,iklass,add  ] localSleep(()V): flush obsolete method @4 in version @0
[3.295s][trace][redefine,class,iklass,add  ] scratch class added; one of its methods is on_stack.
[3.328s][trace][redefine,class,iklass,purge] Class unloading: has_previous_versions = true
[3.328s][trace][redefine,class,iklass,purge] RedefineRunningMethods_B: previous versions
[3.328s][trace][redefine,class,iklass,purge] previous version 0x000000080113b178 is alive
[3.328s][trace][redefine,class,iklass,purge] previous version 0x000000080113ab30 is alive
[3.328s][trace][redefine,class,iklass,purge] previous version stats: live=2, deleted=0
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/nmethod.cpp:3249
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S94905/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/5ecec92a-60d9-4c7e-9510-cf8bac54efe9/runs/863af1fa-8803-489d-829f-2e54063e48d5/workspace/open/src/hotspot/share/code/nmethod.cpp:3249), pid=3351329, tid=3351333
#  assert(db != __null && !db->is_adapter_blob()) failed: must use stub!
#
# JRE version: Java(TM) SE Runtime Environment (20.0+19) (fastdebug build 20-ea+19-1326)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-ea+19-1326, mixed mode, sharing, tiered, compressed class ptrs, z gc, linux-aarch64)
# Problematic frame:
# V  [libjvm.so+0x1448424][thread 3351361 also had an error]
  DirectNativeCallWrapper::verify_resolve_call(unsigned char*) const+0x54
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S94889/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6ca99daf-28b6-456d-84e1-2511b8bcab33/runs/e3985560-a056-4d43-82fc-761d1adaf63f/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/scratch/3/core.3351329)
#
Unsupported internal testing APIs have been used.

# An error report file with more information is saved as:
# /opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S94889/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6ca99daf-28b6-456d-84e1-2511b8bcab33/runs/e3985560-a056-4d43-82fc-761d1adaf63f/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/scratch/3/hs_err_pid3351329.log
[thread 3351359 also had an error]
[thread 3351357 also had an error]
[thread 3351394 also had an error]
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
];
 stderr: [java version "20-ea" 2023-03-21
Java(TM) SE Runtime Environment (fastdebug build 20-ea+19-1326)
Java HotSpot(TM) 64-Bit Server VM (fastdebug build 20-ea+19-1326, mixed mode, sharing)
]
 exitValue = 134

java.lang.RuntimeException: Hotspot crashed
	at jdk.test.lib.cds.CDSTestUtils.checkCommonExecExceptions(CDSTestUtils.java:320)
	at jdk.test.lib.cds.CDSTestUtils.checkExec(CDSTestUtils.java:473)
	at jdk.test.lib.cds.CDSTestUtils.checkExec(CDSTestUtils.java:460)
	at RedefineRunningMethods_Shared.main(RedefineRunningMethods_Shared.java:78)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
	at java.base/java.lang.Thread.run(Thread.java:1591)

JavaTest Message: Test threw exception: java.lang.RuntimeException
JavaTest Message: shutting down test

result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: Hotspot crashed


Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x0000fffd08054ce0):  WorkerThread "ZWorker#0" [stack: 0x0000fffd0cbe0000,0x0000fffd0cde0000] [id=3351333]

Stack: [0x0000fffd0cbe0000,0x0000fffd0cde0000],  sp=0x0000fffd0cdde5c0,  free space=2041k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1448424]  DirectNativeCallWrapper::verify_resolve_call(unsigned char*) const+0x54  (nmethod.cpp:3249)
V  [libjvm.so+0x9a4778]  CompiledIC::is_call_to_interpreted() const+0x98  (compiledIC.cpp:353)
V  [libjvm.so+0x9a71d4]  CompiledIC::verify()+0x54  (compiledIC.cpp:691)
V  [libjvm.so+0x9ad8d0]  CompiledMethod::cleanup_inline_caches_impl(bool, bool)+0x1f0  (compiledIC.hpp:303)
V  [libjvm.so+0x9ae738]  CompiledMethod::unload_nmethod_caches(bool)+0xc8  (compiledMethod.cpp:550)
V  [libjvm.so+0x194ae4c]  ZNMethodUnlinkClosure::do_nmethod(nmethod*)+0xbc  (zNMethod.cpp:309)
V  [libjvm.so+0x194d0ac]  ZNMethodTableIteration::nmethods_do(NMethodClosure*)+0x8c  (zNMethodTableIteration.cpp:72)
V  [libjvm.so+0x194aa48]  ZNMethodUnlinkTask::work()+0x28  (zNMethod.cpp:338)
V  [libjvm.so+0x197f708]  ZTask::Task::work(unsigned int)+0x28  (zTask.cpp:34)
V  [libjvm.so+0x18f3678]  WorkerThread::run()+0x98  (workerThread.cpp:69)
V  [libjvm.so+0x17af638]  Thread::call_run()+0xf8  (thread.cpp:224)
V  [libjvm.so+0x14c4824]  thread_native_entry(Thread*)+0x100  (os_linux.cpp:710)
C  [libpthread.so.0+0x7908]  start_thread+0x188


This is a ZGC WorkerThread calling compiler code and failing
an assertion. I'm starting this bug off in hotspot/gc for initial triage.
Comments
Good idea [~dcubed]. I will do that.
20-12-2022

[~eosterlund] - Perhaps this bug should be closed as a duplicate of JDK-8293584 instead of being closed as "Not an Issue". Or you can closed it as "Not Reproducible".
19-12-2022

Fixed by https://bugs.openjdk.org/browse/JDK-8293584
19-12-2022

As the symptoms of the crash reported here matches exactly those reported under https://bugs.openjdk.org/browse/JDK-8293584 which was not fixed when this was filed, it seems very plausible that it is the same issue. It has not been reproduced since it was fixed.
19-12-2022

CPU23_01-defer-request: We do not have any further understanding at this point about this issue, and from as far as we can tell it's not a showstopper, so requesting deferral from that cpu release
30-11-2022

Since this is not unlikely to be caused by JDK-8293584, I'm linking this bug as blocked by that bug.
15-11-2022

It's worth mentioning that there is a known race between class redefinition and concurrent class unloading that I filed under JDK-8293584. So when crashing in JDK20 with ZGC when doing lots of class redefinition, I have to wonder if there could be a link here. But there is also the metadata relocation clearing issue that Dean mentions from JDK-8296440. When I wrote that code in JDK 12, we didn't support AArch64, so I did not consider what AArch64 should do there.
08-11-2022

Since this happens on sparc and aarch64, I think it could be related to how those platforms generate code for static call stubs. Unlike x86, which puts the metadata in the instruction, sparc and aarch64 use oop_recorder()->allocate_metadata_index() to allocate a metadata slot AND they put the metadata in one or more instructions. JDK-8296440 points out that fix_metadata_relocation() does nothing on most platforms, and then speculates that some code can be removed as redundant. But these crashes make me thing that there may be situations, like class redefinition, where these duplicate pieces of metadata need to be kept consistent.
07-11-2022

Not unlikely to be related to JDK-8294538
11-10-2022