JDK-8317262 : LockStack::contains(oop) fails "assert(t->is_Java_thread()) failed: incorrect cast to JavaThread"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 21,22,repo-lilliput-17
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux,windows
  • CPU: x86_64
  • Submitted: 2023-09-28
  • Updated: 2024-04-08
  • Resolved: 2023-10-15
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 21 JDK 22
21.0.2Fixed 22 b20Fixed
Related Reports
Relates :  
JDK-8315880 - Change LockingMode default from LM_LEGACY to LM_LIGHTWEIGHT
Relates :  
JDK-8291555 - Implement alternative fast-locking scheme
Relates :  
JDK-8316538 - runtime/handshake/MixedHandshakeWalkStackTest.java crashes with JFR
Description
The following test failed in the JDK22 CI:

runtime/handshake/MixedHandshakeWalkStackTest.java

Here's a snippet from the log file:

"MainThread" #20 [3778977] prio=5 os_prio=0 cpu=2797.22ms elapsed=25.96s tid=0x00007fc5ec32f330 nid=3778977 waiting on condition  [0x00007fc5c2e9c000]
   java.lang.Thread.State: RUNNABLE
Thread: 0x00007fc5ec32f330  [0x39a9a1] State: _running _at_poll_safepoint 0
   JavaThread state: _thread_blocked
	at java.lang.Thread.join(java.base@22-ea/Thread.java:2045)
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/opt/mach5/mesos/work_dir/slaves/cd627e65-f015-4fb1-a1d2-b6c9b8127f98-S202952/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/7ea1439c-6a8e-4755-9db4-5138d6e04957/runs/54ab28d1-851a-402f-960a-cd3e8bcfc6aa/workspace/open/src/hotspot/share/runtime/javaThread.hpp:980), pid=3778833, tid=3778844
#  assert(t->is_Java_thread()) failed: incorrect cast to JavaThread
#
# JRE version: Java(TM) SE Runtime Environment (22.0+17) (fastdebug build 22-ea+17-1331)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 22-ea+17-1331, compiled mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x1743fd5]  LockStack::contains(oop) const+0x325
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/cd627e65-f015-4fb1-a1d2-b6c9b8127f98-S200905/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/af57202a-9f78-4b5d-a63b-ece9be24e78a/runs/93c402bc-fad1-4370-a9d0-0ac8b729f0c9/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/scratch/2/core.3778833)
#
Unsupported internal testing APIs have been used.

# An error report file with more information is saved as:
# /opt/mach5/mesos/work_dir/slaves/cd627e65-f015-4fb1-a1d2-b6c9b8127f98-S200905/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/af57202a-9f78-4b5d-a63b-ece9be24e78a/runs/93c402bc-fad1-4370-a9d0-0ac8b729f0c9/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/scratch/2/hs_err_pid3778833.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
----------System.err:(0/0)----------
----------rerun:(46/8741)*----------


Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x00007fc5ec2dc530):  VMThread "VM Thread"          [id=3778844, stack(0x00007fc5c3879000,0x00007fc5c3979000) (1024K)] _threads_hazard_ptr=0x00007fc538001630, _nested_threads_hazard_ptr_cnt=0

Stack: [0x00007fc5c3879000,0x00007fc5c3979000],  sp=0x00007fc5c39761e0,  free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1743fd5]  LockStack::contains(oop) const+0x325  (javaThread.hpp:980)
V  [libjvm.so+0x185b6b5]  javaVFrame::print_lock_info_on(outputStream*, int)+0x9c5  (objectMonitor.inline.hpp:39)
V  [libjvm.so+0xec281e]  JavaThread::print_stack_on(outputStream*)+0x1de  (javaThread.cpp:1755)
V  [libjvm.so+0x18bde8f]  WB_HandshakeWalkStack::TraceSelfClosure::do_thread(Thread*)+0x7f  (whitebox.cpp:2237)
V  [libjvm.so+0xda58c6]  HandshakeOperation::do_handshake(JavaThread*)+0x46  (handshake.cpp:326)
V  [libjvm.so+0xda5e6a]  HandshakeState::try_process(HandshakeOperation*) [clone .part.0]+0xfa  (handshake.cpp:671)
V  [libjvm.so+0xda8263]  VM_HandshakeAllThreads::doit()+0x443  (handshake.cpp:651)
V  [libjvm.so+0x1888fc9]  VM_Operation::evaluate()+0x139  (vmOperations.cpp:71)
V  [libjvm.so+0x18a9c5b]  VMThread::evaluate_operation(VM_Operation*)+0x10b  (vmThread.cpp:281)
V  [libjvm.so+0x18aa61e]  VMThread::inner_execute(VM_Operation*)+0x20e  (vmThread.cpp:435)
V  [libjvm.so+0x18aa9b4]  VMThread::loop()+0x84  (vmThread.cpp:502)
V  [libjvm.so+0x18aaac2]  VMThread::run()+0x92  (vmThread.cpp:175)
V  [libjvm.so+0x17a338a]  Thread::call_run()+0xba  (thread.cpp:220)
V  [libjvm.so+0x14a89ca]  thread_native_entry(Thread*)+0x12a  (os_linux.cpp:786)


The test task's JVM args are:
-Xcomp -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:+TieredCompilation -XX:-DoEscapeAnalysis
Comments
A pull request was submitted for review. URL: https://git.openjdk.org/lilliput-jdk17u/pull/75 Date: 2024-04-08 18:32:54 +0000
08-04-2024
[jdk21u-fix-request] Approval Request from Aleksey Shipilëv Clean backport to fix the trivial bug in new light-weight locking. This code path is disabled by default in JDK 21u, but the bug would manifest if any user experimentally enables it.
17-10-2023
A pull request was submitted for review. URL: https://git.openjdk.org/jdk21u/pull/262 Date: 2023-10-17 09:02:25 +0000
17-10-2023
Changeset: 4ea1b99c Author: David Holmes <dholmes@openjdk.org> Date: 2023-10-15 23:34:16 +0000 URL: https://git.openjdk.org/jdk/commit/4ea1b99c1a6efe144af381ea538f93718e9baf74
15-10-2023
Trying to understand what may have changed here. The WB_HandshakeWalkStack method has been around since handshakes were introduced and has always assumed/expected to only be executed by a JavaThread. So what changed to cause it to be executed by the VMThread? And was that always a possibility? The test itself has not been changed in over a year too. Correction: "has always assumed/expected to only be executed by a JavaThread" should be "has always assumed/expected to only be applied to a JavaThread". I misread the code, the current thread is not used whilst executing the actual handshake.
05-10-2023
A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/16047 Date: 2023-10-05 01:47:03 +0000
05-10-2023
Thanks [~pchilanomate] - for some reason JBS didn't show me your comment until after I had rediscovered what you wrote.
04-10-2023
Unclear exactly why this triggered now but the bug is here: inline bool LockStack::is_owning_thread() const { JavaThread* thread = JavaThread::current(); bool is_owning = &thread->lock_stack() == this; assert(is_owning == (get_thread() == thread), "is_owning sanity"); return is_owning; } If the current thread is not a JavaThread we should be returning false. This is called from: inline bool LockStack::contains(oop o) const { verify("pre-contains"); if (!SafepointSynchronize::is_at_safepoint() && !is_owning_thread()) {
04-10-2023
I took a look at this issue. The test is just uncovering a bug in LockStack::is_owning_thread(), which assumes the caller is always a JavaThread, that we have probably not seen before because the condition to trigger it is too rare. The VMThread (or any non-JavaThread) needs to call JavaThread::print_stack_on() on a JT which in the top frame owns a monitor that is inflated and the owner is marked as anonymous. This MixedHandshakeWalkStackTest.java test uses a handshake that does call JavaThread::print_stack_on(), so it just happened that we hit some handshake all operation by the VMThread and it took over that other handshake. Now, how we find an inflated monitor marked as anonymous in the top frame I don't understand, it must be during initialization code, since I don't see any explicit synchronized in the test. I attached a simple test that reproduces the issue.
04-10-2023
ILW = HLM = P3
03-10-2023
Yeah the WB handshake stack walking code is breaking invariants by allowing, in this case, the VMThread to do stuff we only expected JavaThreads to do.
29-09-2023
There's a recent different crash with the same test: JDK-8316538 runtime/handshake/MixedHandshakeWalkStackTest.java crashes with JFR
28-09-2023