JDK-8303086 : SIGSEGV in JavaThread::is_interp_only_mode()
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jvmti
  • Affected Version: 21
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: os_x
  • CPU: aarch64
  • Submitted: 2023-02-22
  • Updated: 2023-07-25
  • Resolved: 2023-06-30
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 21 JDK 22
21Fixed 22 b05Fixed
Related Reports
Relates :  
JDK-8311218 - fatal error: stuck in JvmtiVTMSTransitionDisabler::VTMS_transition_disable
Relates :  
JDK-8311177 - Switching to interpreter only mode in carrier thread can lead to crashes
Description
The following test crashed with a SIGSEGV in JavaThread::is_interp_only_mode():

serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

Here's snippets from the hs_err_pid file:

#  SIGSEGV (0xb) at pc=0x0000000109ebf5cc, pid=5235, tid=25603
#
# JRE version: Java(TM) SE Runtime Environment (21.0) (slowdebug build 21-internal-LTS-2023-02-16-1211495.dcubed...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (slowdebug 21-internal-LTS-2023-02-16-1211495.dcubed..., mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# V  [libjvm.dylib+0xebf5cc]  JavaThread::is_interp_only_mode()+0x14

<snip>

---------------  T H R E A D  ---------------

Current thread (0x0000000133810a10):  JavaThread "JVMTI agent thread" daemon [_thread_in_vm, id=25603, stack(0x000000016d998000,0x000000016db9b000)] _threads_hazard_ptr=0x0000000125558250, _nested_threads_hazard_ptr_cnt=1, _nested_threads_hazard_ptr=0x0000000125558250

Stack: [0x000000016d998000,0x000000016db9b000],  sp=0x000000016db9a6e0,  free space=2057k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.dylib+0xebf5cc]  JavaThread::is_interp_only_mode()+0x14
V  [libjvm.dylib+0xb6a700]  JvmtiThreadState::is_interp_only_mode()+0x44
V  [libjvm.dylib+0xb6c660]  JvmtiEventControllerPrivate::recompute_thread_enabled(JvmtiThreadState*)+0x164
V  [libjvm.dylib+0xb6c91c]  JvmtiEventControllerPrivate::recompute_enabled()+0x1f0
V  [libjvm.dylib+0xb6df20]  JvmtiEventControllerPrivate::set_user_enabled(JvmtiEnvBase*, JavaThread*, Handle, jvmtiEvent, bool)+0x1a4
V  [libjvm.dylib+0xb6e570]  JvmtiEventController::set_user_enabled(JvmtiEnvBase*, JavaThread*, oopDesc*, jvmtiEvent, bool)+0xfc
V  [libjvm.dylib+0xb52588]  JvmtiEnv::SetEventNotificationMode(jvmtiEventMode, jvmtiEvent, _jobject*, ...)+0x1a4
V  [libjvm.dylib+0xafad3c]  jvmti_SetEventNotificationMode(_jvmtiEnv*, jvmtiEventMode, jvmtiEvent, _jobject*, ...)+0x140
C  [libGetStackTraceSuspendedStress.dylib+0x3b18]  _jvmtiEnv::SetEventNotificationMode(jvmtiEventMode, jvmtiEvent, _jobject*, ...)+0x3c
C  [libGetStackTraceSuspendedStress.dylib+0x5e50]  check_vthread_consistency_suspended(_jvmtiEnv*, JNIEnv_*, _jobject*)+0xc0
C  [libGetStackTraceSuspendedStress.dylib+0x4d80]  agentProc(_jvmtiEnv*, JNIEnv_*, void*)+0x19c
C  [libGetStackTraceSuspendedStress.dylib+0x545c]  agent_thread_wrapper(_jvmtiEnv*, JNIEnv_*, void*)+0x54
V  [libjvm.dylib+0xb801d8]  JvmtiAgentThread::call_start_function()+0x74
V  [libjvm.dylib+0xb8010c]  JvmtiAgentThread::start_function_wrapper(JavaThread*, JavaThread*)+0x60
V  [libjvm.dylib+0x90be18]  JavaThread::thread_main_inner()+0x108
V  [libjvm.dylib+0x90bd04]  JavaThread::run()+0x124
V  [libjvm.dylib+0xff82b0]  Thread::call_run()+0x154
V  [libjvm.dylib+0xdb6ed4]  thread_native_entry(Thread*)+0x138
C  [libsystem_pthread.dylib+0x7878]  _pthread_start+0x140

siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x0000000000000560
Comments
A pull request was submitted for review. URL: https://git.openjdk.org/jdk21/pull/96 Date: 2023-07-05 19:33:16 +0000
05-07-2023
Changeset: 971c2efb Author: Serguei Spitsyn <sspitsyn@openjdk.org> Date: 2023-06-30 20:58:15 +0000 URL: https://git.openjdk.org/jdk/commit/971c2efb698065c65dcf7373d8c3027f58d5f503
30-06-2023
A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/14728 Date: 2023-06-30 11:27:58 +0000
30-06-2023
Here's the logs from my jdk-21+26 stress run sighting on macosx-aarch64: $ unzip -l jdk-21+26_macosx-aarch64.8303086.zip Archive: jdk-21+26_macosx-aarch64.8303086.zip Length Date Time Name --------- ---------- ----- ---- 23644 06-09-2023 17:22 jdk-21+26_1/failures.macosx-aarch64/GetStackTraceSuspendedStressTest.jtr.slowdebug 111383 06-09-2023 17:22 jdk-21+26_1/failures.macosx-aarch64/hs_err_pid55884.log --------- ------- 135027 2 files
12-06-2023
Here's the logs from my jdk-21+25 stress run sighting on macosx-aarch64: $ unzip -l jdk-21+25_macosx-aarch64.8303086.zip Archive: jdk-21+25_macosx-aarch64.8303086.zip Length Date Time Name --------- ---------- ----- ---- 23156 06-03-2023 05:48 jdk-21+25_2/failures.macosx-aarch64/GetStackTraceSuspendedStressTest.jtr.slowdebug 111617 06-03-2023 05:48 jdk-21+25_2/failures.macosx-aarch64/hs_err_pid54687.log --------- ------- 134773 2 files
05-06-2023
ILW=MMH=P3
02-06-2023
Here's the logs from my jdk-21+24 stress run sighting on macosx-aarch64: $ unzip -l jdk-21+24_macosx-aarch64.8303086.zip Archive: jdk-21+24_macosx-aarch64.8303086.zip Length Date Time Name --------- ---------- ----- ---- 23249 05-26-2023 04:46 jdk-21+24_1/failures.macosx-aarch64/GetStackTraceSuspendedStressTest.jtr.slowdebug 111505 05-26-2023 04:46 jdk-21+24_1/failures.macosx-aarch64/hs_err_pid69175.log --------- ------- 134754 2 files
30-05-2023
[~dcubed] Unfortunately JDK-8307365 will not fix this issue. The problem here is that calling SetEventNotificationMode() targeting a single thread will just disable mount/unmount transitions for that particular thread, but when we later call recompute_enable() we will loop over all JvmtiThreadStates. The issue with that is that in the transitions the _thread field of a JvmtiThreadState can change from non-null to null (or the other way around). So in the crashing case we read a non-null value of _thread for the JvmtiThreadState of some thread that is transitioning, but then when we try to dereference it we first read it again and now the value is null so we crash in the dereference. This is the code where we crash: bool is_interp_only_mode() { return _thread == nullptr ? _saved_interp_only_mode != 0 : _thread->is_interp_only_mode(); } The crash actually depends on the compiler generating the extra load instead of using the first one. That's why we are only seeing this crash in slowdebug mode. You can reproduce the crash easily in slowdebug mode using this patch: diff --git a/src/hotspot/share/prims/jvmtiThreadState.hpp b/src/hotspot/share/prims/jvmtiThreadState.hpp index 8340a44d142..6136fe08358 100644 --- a/src/hotspot/share/prims/jvmtiThreadState.hpp +++ b/src/hotspot/share/prims/jvmtiThreadState.hpp @@ -259,7 +259,12 @@ class JvmtiThreadState : public CHeapObj<mtInternal> { // Used by the interpreter for fullspeed debugging support bool is_interp_only_mode() { - return _thread == nullptr ? _saved_interp_only_mode != 0 : _thread->is_interp_only_mode(); + if (_thread == nullptr) { + return _saved_interp_only_mode != 0; + } else { + os::naked_short_sleep(1); + return _thread->is_interp_only_mode(); + } } void enter_interp_only_mode(); void leave_interp_only_mode();
22-05-2023
[~pchilanomate]'s fix for the following bug: JDK-8307365 JvmtiStressModule hit SIGSEGV in JvmtiEventControllerPrivate::recompute_thread_enabled is not included in jdk-21+23, but will be included in the jdk-21+24 snapshot. I'm wondering if the fix for JDK-8307365 will also address this bug (JDK-8303086).
22-05-2023
Here's the logs from my jdk-21+23 stress run sighting on macosx-aarch64: $ unzip -l jdk-21+23_macosx-aarch64.8303086.zip Archive: jdk-21+23_macosx-aarch64.8303086.zip Length Date Time Name --------- ---------- ----- ---- 23163 05-21-2023 09:16 jdk-21+23_3/failures.macosx-aarch64/GetStackTraceSuspendedStressTest.jtr.slowdebug 111453 05-21-2023 09:16 jdk-21+23_3/failures.macosx-aarch64/hs_err_pid49648.log --------- ------- 134616 2 files
22-05-2023
I looked at this crash and we are indeed accessing a JvmtiThreadState while the corresponding thread is transitioning. In this test the SetEventNotificationMode() operation is made on a particular vthread, meaning the JvmtiVTMSTransitionDisabler only disables transitions for this vthread. But when we call recompute_enabled() we process all states. One of this states happens to be from a thread that is transitioning so we hit the issue that Dan pointed out: - We read _thread from the JvmtiThreadState and see that is not null - Thread goes through a mount/unmount transition where the call to rebind_to_jvmti_thread_state_of() sets _thread to null. - We read _thread again from the JvmtiThreadState state and crash when dereferencing it. The bug is easily reproducible just by adding a os::naked_short_sleep() call between between the first and second read of _thread. 
There is another place in method JvmtiEventControllerPrivate::recompute_thread_enabled() where it seems this crash could happen: if (state->get_thread() != nullptr) { // The JavaThread for carrier or mounted virtual thread case. // Update the cached value for thread-specific should_post_on_exceptions value. bool should_post_on_exceptions = (any_env_enabled & SHOULD_POST_ON_EXCEPTIONS_BITS) != 0; state->set_should_post_on_exceptions(should_post_on_exceptions); } where set_should_post_on_exceptions() is defined as: inline void JvmtiThreadState::set_should_post_on_exceptions(bool val) { get_thread_or_saved()->set_should_post_on_exceptions_flag(val ? JNI_TRUE : JNI_FALSE); } But that would require the "if (any_env_enabled != was_any_env_enabled)" conditional to be true for other threads besides the one we actually requested the operation on. For this test that never happens. But if it is possible in other cases then we would crash there too.
09-05-2023
Here's the logs from my jdk-21+19 stress run sighting on macosx-aarch64: $ unzip -l jdk-21+19_macosx-aarch64.8303086.zip Archive: jdk-21+19_macosx-aarch64.8303086.zip Length Date Time Name --------- ---------- ----- ---- 23150 04-21-2023 04:49 jdk-21+19_1/failures.macosx-aarch64/GetStackTraceSuspendedStressTest.jtr.slowdebug 109924 04-21-2023 04:49 jdk-21+19_1/failures.macosx-aarch64/hs_err_pid18550.log --------- ------- 133074 2 files
24-04-2023
Here's the logs from my jdk-21+18 stress run sighting on macosx-aarch64: $ unzip -l jdk-21+18_macosx-aarch64.8303086.zip Archive: jdk-21+18_macosx-aarch64.8303086.zip Length Date Time Name --------- ---------- ----- ---- 23772 04-14-2023 00:46 jdk-21+18_1/failures.macosx-aarch64/GetStackTraceSuspendedStressTest.jtr.slowdebug 110690 04-14-2023 00:46 jdk-21+18_1/failures.macosx-aarch64/hs_err_pid9574.log --------- ------- 134462 2 files
17-04-2023
Here's the logs from my jdk-21+17 stress run sighting on linux-x64: $ unzip -l jdk-21+17_linux-x64.8303086.zip Archive: jdk-21+17_linux-x64.8303086.zip Length Date Time Name --------- ---------- ----- ---- 22475 2023-04-07 02:39 jdk-21+17_1/failures.linux-x86_64/GetStackTraceSuspendedStressTest.jtr.slowdebug --------- ------- 22475 1 file
10-04-2023
[~sspitsyn] - I think you mean a different bug. JDK-8299240 is fixed in jdk-21+11.
07-03-2023
Thanks, Dan. Let's see if it is failing in the jdk main line. I'm suspecting that this can be fixed by the https://bugs.openjdk.org/browse/JDK-8299240 .
02-03-2023
> Are there any links to these stress test runs? These stress test runs are done on machines in my lab so there are no usable links. However, I did include zip files containing the logs and the hs_err_pid files from the runs. I did not see the failure in jdk-21+9 stress testing. However, I suspect that the failure mode can be intermittent.
01-03-2023
Dan, thank you for great analysis as it helps a lot! Is this started failing in the jdk-21+10 or it was also observed earlier? I wonder if it was a regression from this fix integration: https://bugs.openjdk.org/browse/JDK-8298853 : JvmtiVTMSTransitionDisabler should support disabling one virtual thread transitions VTMS transitions of the target thread are supposed to be disabled because the JVMTI code is executed in a context of JvmtiVTMSTransitionDisabler: JvmtiEnv::SetEventNotificationMode(jvmtiEventMode mode, jvmtiEvent event_type, jthread event_thread, ...) { . . . . . . JvmtiVTMSTransitionDisabler disabler(event_thread); <== There is a disabler here! if (event_thread == nullptr) { . . . . . . JvmtiEventController::set_user_enabled(this, nullptr, (oop) nullptr, event_type, enabled); } else { // We have a specified event_thread. ThreadsListHandle tlh; . . . . . . JvmtiEventController::set_user_enabled(this, java_thread, thread_obj, event_type, enabled); <== This call is executed Your suggestion fix is good as a work around. However, it seems, there is a bug somewhere in the implementation which would be nice to identify. Also, I wonder if this problem was fixed with the integration of this one: https://bugs.openjdk.org/browse/JDK-8299240 : rank of JvmtiVTMSTransition_lock can be safepoint Are there any links to these stress test runs?
01-03-2023
Here's the logs from my jdk-21+11 stress run sighting on macosx-aarch64: $ unzip -l jdk-21+11_macosx-aarch64.8303086.zip Archive: jdk-21+11_macosx-aarch64.8303086.zip Length Date Time Name --------- ---------- ----- ---- 23694 02-25-2023 06:00 jdk-21+11_2/failures.macosx-aarch64/GetStackTraceSuspendedStressTest.jtr.slowdebug 110824 02-25-2023 06:00 jdk-21+11_2/failures.macosx-aarch64/hs_err_pid27685.log --------- ------- 134518 2 files
28-02-2023
Sorry Dan tl;dr :)
23-02-2023
[~dholmes] - please see the proposed fix at the end of this comment: https://bugs.openjdk.org/browse/JDK-8303086?focusedCommentId=14562326&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14562326 > _thread should be read only once in this kind of check. That's exactly what I proposed...
23-02-2023
A racy is_interp_mode is also suspected in JDK-8302351 > return _thread == nullptr ? _saved_interp_only_mode != 0 : _thread->is_interp_only_mode(); _thread should be read only once in this kind of check.
23-02-2023
It looks like is_interp_only_mode was recently changed during the work to convert NULL -> nullptr, but the bug exists before that: 9583e3657e43 (Alan Bateman 2022-05-07 08:06:16 +0000 236) bool is_interp_only_mode() { 9583e3657e43 (Alan Bateman 2022-05-07 08:06:16 +0000 237) return _thread == NULL ? _saved_interp_only_mode != 0 : _thread->is_interp_only_mode(); 9583e3657e43 (Alan Bateman 2022-05-07 08:06:16 +0000 238) } 9583e3657e43 is the original integration of JEP-425.
22-02-2023
So here's the code that sets the JvmtiThreadState::_thread field: void JvmtiThreadState::set_thread(JavaThread* thread) { _thread_saved = nullptr; // Common case. if (!_is_virtual && thread == nullptr) { // Save JavaThread* if carrier thread is being detached. _thread_saved = _thread; } _thread = thread; } Based on the above code and what I found in the core file, I think the carrier thread was detached from JavaThread* 0x0000000135029410 at the same time that the crashing code was calling: JavaThread::is_interp_only_mode() I don't know if asynchronous carrier thread unmounting is allowed, but if it is, then this code is bad: // Used by the interpreter for fullspeed debugging support bool is_interp_only_mode() { return _thread == nullptr ? _saved_interp_only_mode != 0 : _thread->is_interp_only_mode(); } because we can see a non-nullptr value here: _thread == nullptr and then try to use then non-nullptr value here: _thread->is_interp_only_mode() and crash. A better way to write this function is: // Used by the interpreter for fullspeed debugging support bool is_interp_only_mode() { JavaThread* jt = _thread; return jt == nullptr ? _saved_interp_only_mode != 0 : jt->is_interp_only_mode(); }
22-02-2023
Here's the backtrace from lldb: (lldb) bt warning: could not execute support code to read Objective-C class data in the process. This may reduce the quality of type information available. * thread #22, stop reason = signal SIGSTOP * frame #0: 0x00000001a2a6ce28 libsystem_kernel.dylib`__pthread_kill + 8 frame #1: 0x00000001a2a9f43c libsystem_pthread.dylib`pthread_kill + 292 frame #2: 0x00000001a29e7454 libsystem_c.dylib`abort + 124 frame #3: 0x0000000109dc37a0 libjvm.dylib`os::abort(dump_core=true, siginfo=0x000000016db9a290, context=0x000000016db9a2f8) at os_posix.cpp:2014:5 frame #4: 0x000000010a0a58d0 libjvm.dylib`VMError::report_and_die(id=11, message=0x0000000000000000, detail_fmt="%s", detail_args="\x9eL\U0000001a\n\U00000001", thread=0x0000000133810a10, pc="\taE\xb9?\U00000001", siginfo=0x000000016db9a290, context=0x000000016db9a2f8, filename=0x0000000000000000, lineno=0, size=0) at vmError.cpp:1687:7 frame #5: 0x000000010a0a49fc libjvm.dylib`VMError::report_and_die(thread=0x0000000133810a10, sig=11, pc="\taE\xb9?\U00000001", siginfo=0x000000016db9a290, context=0x000000016db9a2f8, detail_fmt="%s") at vmError.cpp:1339:3 frame #6: 0x000000010a0a5974 libjvm.dylib`VMError::report_and_die(thread=0x0000000133810a10, sig=11, pc="\taE\xb9?\U00000001", siginfo=0x000000016db9a290, context=0x000000016db9a2f8) at vmError.cpp:1345:3 frame #7: 0x0000000109ee0d40 libjvm.dylib`::JVM_handle_bsd_signal(sig=11, info=0x000000016db9a290, ucVoid=0x000000016db9a2f8, abort_if_unrecognized=1) at signals_posix.cpp:655:5 frame #8: 0x0000000109ee22c0 libjvm.dylib`javaSignalHandler(sig=11, info=0x000000016db9a290, context=0x000000016db9a2f8) at signals_posix.cpp:666:9 frame #9: 0x00000001a2aeac44 libsystem_platform.dylib`_sigtramp + 56 frame #10: 0x0000000109b6a700 libjvm.dylib`JvmtiThreadState::is_interp_only_mode(this=0x0000000000000000) at jvmtiThreadState.hpp:241:74 frame #11: 0x0000000109b6a700 libjvm.dylib`JvmtiThreadState::is_interp_only_mode(this=0x00000001254507c0) at jvmtiThreadState.hpp:241:74 frame #12: 0x0000000109b6c660 libjvm.dylib`JvmtiEventControllerPrivate::recompute_thread_enabled(state=0x00000001254507c0) at jvmtiEventController.cpp:593:31 frame #13: 0x0000000109b6c91c libjvm.dylib`JvmtiEventControllerPrivate::recompute_enabled() at jvmtiEventController.cpp:668:33 frame #14: 0x0000000109b6df20 libjvm.dylib`JvmtiEventControllerPrivate::set_user_enabled(env=0x000000013430d670, thread=0x0000000135808e10, thread_oop_h=Handle @ 0x000000016db9a9f8, event_type=JVMTI_EVENT_SINGLE_STEP, enabled=false) at jvmtiEventController.cpp:907:3 frame #15: 0x0000000109b6e570 libjvm.dylib`JvmtiEventController::set_user_enabled(env=0x000000013430d670, thread=0x0000000135808e10, thread_oop=0x00000007fa678b98, event_type=JVMTI_EVENT_SINGLE_STEP, enabled=false) at jvmtiEventController.cpp:1060:5 frame #16: 0x0000000109b52588 libjvm.dylib`JvmtiEnv::SetEventNotificationMode(this=0x000000013430d670, mode=JVMTI_DISABLE, event_type=JVMTI_EVENT_SINGLE_STEP, event_thread=0x00000001254a1f48) at jvmtiEnv.cpp:603:5 frame #17: 0x0000000109afad3c libjvm.dylib`jvmti_SetEventNotificationMode(env=0x000000013430d670, mode=JVMTI_DISABLE, event_type=JVMTI_EVENT_SINGLE_STEP, event_thread=0x00000001254a1f48) at jvmtiEnter.cpp:5321:22 frame #18: 0x0000000104bdfb18 libGetStackTraceSuspendedStress.dylib`_jvmtiEnv::SetEventNotificationMode(this=0x000000013430d670, mode=JVMTI_DISABLE, event_type=JVMTI_EVENT_SINGLE_STEP, event_thread=0x00000001254a1f48) at jvmti.h:2522:12 frame #19: 0x0000000104be1e50 libGetStackTraceSuspendedStress.dylib`check_vthread_consistency_suspended(jvmti=0x000000013430d670, jni=0x0000000133810d38, vthread=0x00000001254a1f48) at libGetStackTraceSuspendedStress.cpp:131:16 frame #20: 0x0000000104be0d80 libGetStackTraceSuspendedStress.dylib`agentProc(jvmti=0x000000013430d670, jni=0x0000000133810d38, arg=0x0000000000000000) at libGetStackTraceSuspendedStress.cpp:206:9 frame #21: 0x0000000104be145c libGetStackTraceSuspendedStress.dylib`agent_thread_wrapper(jvmti_env=0x000000013430d670, agentJNI=0x0000000133810d38, arg=0x0000000000000000) at jvmti_thread.h:179:5 frame #22: 0x0000000109b801d8 libjvm.dylib`JvmtiAgentThread::call_start_function(this=0x0000000133810a10) at jvmtiImpl.cpp:89:5 frame #23: 0x0000000109b8010c libjvm.dylib`JvmtiAgentThread::start_function_wrapper(thread=0x0000000133810a10, __the_thread__=0x0000000133810a10) at jvmtiImpl.cpp:83:14 frame #24: 0x000000010990be18 libjvm.dylib`JavaThread::thread_main_inner(this=0x0000000133810a10) at javaThread.cpp:710:5 frame #25: 0x000000010990bd04 libjvm.dylib`JavaThread::run(this=0x0000000133810a10) at javaThread.cpp:695:3 frame #26: 0x0000000109ff82b0 libjvm.dylib`Thread::call_run(this=0x0000000133810a10) at thread.cpp:224:9 frame #27: 0x0000000109db6ed4 libjvm.dylib`thread_native_entry(thread=0x0000000133810a10) at os_bsd.cpp:572:11 frame #28: 0x00000001a2a9f878 libsystem_pthread.dylib`_pthread_start + 320 Here's the crashing frame and 'this' is nullptr: (lldb) frame select 10 frame #10: 0x0000000109b6a700 libjvm.dylib`JvmtiThreadState::is_interp_only_mode(this=0x0000000000000000) at jvmtiThreadState.hpp:241:74 238 239 // Used by the interpreter for fullspeed debugging support 240 bool is_interp_only_mode() { -> 241 return _thread == nullptr ? _saved_interp_only_mode != 0 : _thread->is_interp_only_mode(); 242 } 243 void enter_interp_only_mode(); 244 void leave_interp_only_mode(); (lldb) print this (JvmtiThreadState *) $0 = nullptr That's not good so let's check the previous frame: (lldb) up frame #11: 0x0000000109b6a700 libjvm.dylib`JvmtiThreadState::is_interp_only_mode(this=0x00000001254507c0) at jvmtiThreadState.hpp:241:74 238 239 // Used by the interpreter for fullspeed debugging support 240 bool is_interp_only_mode() { -> 241 return _thread == nullptr ? _saved_interp_only_mode != 0 : _thread->is_interp_only_mode(); 242 } 243 void enter_interp_only_mode(); 244 void leave_interp_only_mode(); (lldb) print this (JvmtiThreadState *) $1 = 0x00000001254507c0 (lldb) print *this (JvmtiThreadState) $2 = { _thread = nullptr _thread_saved = 0x0000000135029410 _thread_oop_h = { _obj = 0x0000000134048728 } _jvmti_event_queue = nullptr _is_virtual = false _hide_single_stepping = false _pending_interp_only_mode = false _pending_step_for_popframe = false _pending_step_for_earlyret = false _hide_level = 0 _exception_state = ES_CLEARED _class_being_redefined = nullptr _class_load_kind = jvmti_class_load_kind_load _classes_being_redefined = nullptr _cur_stack_depth = -99 _saved_interp_only_mode = 0 _thread_event_enable = { _event_enabled = (_enabled_bits = 0, _init_guard = JEE_INIT_GUARD) } _head_env_thread_state = 0x00000001254508a0 _next = 0x0000000125450670 _prev = 0x0000000125450910 _dynamic_code_event_collector = nullptr _vm_object_alloc_event_collector = nullptr _sampled_object_alloc_event_collector = nullptr _the_class_for_redefinition_verification = nullptr _scratch_class_for_redefinition_verification = nullptr _debuggable = true _earlyret_state = 0 _earlyret_tos = ilgl _earlyret_value = (z = '\0', b = '\0', c = 0, s = 0, i = 0, j = 0, f = 0, d = 0, l = 0x0000000000000000) _earlyret_oop = nullptr } Okay so we have a JvmtiThreadState 'this', but the _thread field is nullptr so when we called JavaThread::is_interp_only_mode() using that _thread field we got a nullptr 'this' value. Let's go up another frame: (lldb) up frame #12: 0x0000000109b6c660 libjvm.dylib`JvmtiEventControllerPrivate::recompute_thread_enabled(state=0x00000001254507c0) at jvmtiEventController.cpp:593:31 590 } 591 // compute interp_only mode 592 bool should_be_interp = (any_env_enabled & INTERP_EVENT_BITS) != 0 || has_frame_pops; -> 593 bool is_now_interp = state->is_interp_only_mode(); 594 595 if (should_be_interp != is_now_interp) { 596 if (should_be_interp) { Let's see the code for is_interp_only_mode(): bool is_interp_only_mode() { return _thread == nullptr ? _saved_interp_only_mode != 0 : _thread->is_interp_only_mode(); } Hmm... if the _thread field is nullptr, then we're supposed to return: _saved_interp_only_mode != 0 but the crash stack says that we called: _thread->is_interp_only_mode() instead and crashed with SIGSEGV. Now I have to wonder if the JvmtiThreadState::_thread field can change while this code is executing... Let's try to figure out which thread we had and perhaps this value from JvmtiThreadState will do the trick: _thread_saved = 0x0000000135029410 (lldb) print *(JavaThread*)state->_thread_saved (JavaThread) $5 = { Thread = { ThreadShadow = { _pending_exception = nullptr _exception_file = 0x0000000000000000 _exception_line = 0 } _nmethod_disarmed_guard_value = 1 _gc_data = { [0] = 0 [1] = 8192 [2] = 0 [3] = 17433981653976477952 [4] = 0 [5] = 2048 [6] = 0 [7] = 5169760768 [8] = 17433981653976478193 [9] = 17433981653976478193 [10] = 17433981653976478193 [11] = 17433981653976478193 [12] = 17433981653976478193 [13] = 17433981653976478193 [14] = 17433981653976478193 [15] = 17433981653976478193 [16] = 17433981653976478193 [17] = 17433981653976478193 [18] = 17433981653976478193 } _threads_hazard_ptr = nullptr _threads_list_ptr = nullptr _nested_threads_hazard_ptr_cnt = 0 _suspendible_thread = false _last_handle_mark = 0x000000016e7e29e8 _threads_do_token = 0 _rcu_counter = 0 _missed_ic_stub_refill_verifier = nullptr _skip_gcalot = false _tlab = { _start = 0x00000007fa1ec098 _top = 0x00000007fa1f0e10 _pf_top = 0x00000007fa1ec0c0 _end = 0x00000007fa1ffc00 _allocation_end = 0x00000007fa1ffc00 _desired_size = 60293 _refill_waste_limit = 942 _allocated_before_last_gc = 0 _bytes_since_last_sample_point = 0 _number_of_refills = 3 _refill_waste = 261 _gc_waste = 0 _slow_allocations = 0 _allocated_size = 80676 _allocation_fraction = (_average = 0.999998033, _sample_count = 1, _weight = 35, _is_old = false, _last_sample = 0.999998033) } _allocated_bytes = 561552 _heap_sampler = (_bytes_until_sample = 800497) _statistical_info = (_start_time_stamp = 1277841645146583, _define_class_count = 10) _jfr_thread_local = { _java_event_writer = nullptr _java_buffer = nullptr _native_buffer = nullptr _shelved_buffer = nullptr _load_barrier_buffer_epoch_0 = nullptr _load_barrier_buffer_epoch_1 = nullptr _checkpoint_buffer_epoch_0 = nullptr _checkpoint_buffer_epoch_1 = nullptr _stackframes = nullptr _dcmd_arena = nullptr _thread = { _ptr = nullptr } _vthread_id = 134 _jvm_thread_id = 149 _thread_id_alias = 18446744073709551615 _data_lost = 0 _stack_trace_id = 18446744073709551615 _parent_trace_id = 148 _user_time = 0 _cpu_time = 0 _wallclock_time = 1277841645146750 _stack_trace_hash = 0 _stackdepth = 0 _entering_suspend_flag = 0 _critical_section = 0 _vthread_epoch = 0 _vthread_excluded = false _jvm_thread_excluded = false _vthread = true _dead = false } _current_pending_raw_monitor = nullptr _run_state = RUN _unhandled_oops = 0xf1f1f1f1f1f1f1f1 _osthread = 0x0000000134244e50 _resource_area = 0x0000000134244060 _current_resource_mark = nullptr _handle_area = 0x0000000134244510 _metadata_handles = 0x0000000134244670 _stack_base = 0x000000016e7e3000 "\xdf\xc9C\x937\xbb\x85)" _stack_size = 2109440 _lgrp_id = -1 _owned_locks = nullptr _jvmti_env_iteration_count = 0 _ParkEvent = 0x0000000134244900 _hashStateW = 1851578994 _hashStateX = 222519383 _hashStateY = 1221354648 _hashStateZ = 1553654197 _wx_init = true _wx_state = WXWrite } _on_thread_list = true _threadObj = { _obj = 0x000000013372e260 } _vthread = { _obj = 0x000000013372e268 } _jvmti_vthread = { _obj = 0x000000013372e270 } _scopedValueCache = { _obj = 0x000000013372e278 } _java_call_counter = 1 _anchor = { _last_Java_sp = 0x000000016e7e21f0 _last_Java_pc = 0x000000011b818264 "\xa8" _last_Java_fp = 0x0000000000000000 } _entry_point = 0x0000000109a10a24 (libjvm.dylib`thread_entry(JavaThread*, JavaThread*) at jvm.cpp:2920) _jni_environment = { functions = 0x000000010a4a88c0 } _deopt_mark = nullptr _deopt_nmethod = nullptr _vframe_array_head = nullptr _vframe_array_last = 0x0000000134a2da10 _jvmti_deferred_updates = nullptr _callee_target = 0x000000015807bd40 _vm_result = nullptr _vm_result_2 = nullptr _deferred_card_mark = { _start = nullptr _word_size = 0 } _current_pending_monitor = nullptr _current_pending_monitor_is_from_java = true _current_waiting_monitor = nullptr _active_handles = 0x00000001340588b0 _free_handle_block = 0x0000000134245120 _Stalled = 0 _monitor_chunks = nullptr _suspend_flags = 0 _thread_state = _thread_blocked _poll_data = (_polling_word = 18446744073709551614, _polling_page = 4374495232) _safepoint_state = 0x0000000134244e10 _saved_exception_pc = 0x000000011b81d770 "\x88\xe3A\xf9\xffc(\xebH" _requires_cross_modify_fence = false _no_safepoint_count = 0 _visited_for_critical_count = 0 _terminated = _not_terminated _in_deopt_handler = 0 _doing_unsafe_access = false _do_not_unlock_if_synchronized = false _carrier_thread_suspended = false _is_in_VTMS_transition = true _is_in_tmp_VTMS_transition = false _is_VTMS_transition_disabler = false _jni_attach_state = _not_attaching_via_jni _pending_deoptimization = -1 _pending_monitorenter = false _pending_transfer_to_interpreter = false _in_retryable_allocation = false _pending_failed_speculation = 0 _jvmci = (_implicit_exception_pc = 0x0000000000000000, _alternate_call_target = 0x0000000000000000) _libjvmci_runtime = nullptr _jvmci_counters = 0x0000000000000000 _jvmci_reserved0 = 0 _jvmci_reserved1 = 0 _jvmci_reserved_oop0 = nullptr _stack_overflow_state = { _stack_guard_state = stack_guard_enabled _stack_overflow_limit = 0x000000016e5fc000 "" _reserved_stack_activation = 0x000000016e7e3000 "\xdf\xc9C\x937\xbb\x85)" _shadow_zone_safe_limit = 0x000000016e608000 "" _shadow_zone_growth_watermark = 0x000000016e7defc0 "" _stack_base = 0x000000016e7e3000 "\xdf\xc9C\x937\xbb\x85)" _stack_end = 0x000000016e5e0000 "" } _exception_oop = nullptr _exception_pc = 0x0000000000000000 _exception_handler_pc = 0x0000000000000000 _is_method_handle_return = 0 _jni_active_critical = 0 _pending_jni_exception_check_fn = 0x0000000000000000 _depth_first_number = 0 _popframe_condition = 0 _frames_to_pop_failed_realloc = 0 _cont_entry = 0x000000016e7e22a0 _cont_fastpath = 0x0000000000000000 _cont_fastpath_thread_state = 1 _held_monitor_count = 0 _jni_monitor_count = 0 _stack_watermarks = { _head = nullptr } _handshake = { _handshakee = 0x0000000135029410 _queue = { _first = nullptr } _lock = { Mutex = { _owner = nullptr _lock = { PlatformMutex = { _impl = 0x0000000134244a40 } _impl = 0x0000000134244aa0 } _name = 0x0000000134244af0 "HandshakeState_lock" _allow_vm_block = true _rank = nosafepoint _next = nullptr _last_owner = 0x0000000135029410 _skip_rank_check = false } } _active_handshaker = nullptr _async_exceptions_blocked = false _suspended = false _async_suspend_handshake = false } _popframe_preserved_args = 0x0000000000000000 _popframe_preserved_args_size = 0 _jvmti_thread_state = 0x00000001336aafc0 _interp_only_mode = 0 _should_post_on_exceptions_flag = 0 _thread_stat = 0x0000000134244b20 _parker = { PlatformParker = { _counter = 0 _cur_index = -1 _mutex = { [0] = (__sig = 1297437786, __opaque = char [56] @ 0x0000000166ab4980) } _cond = { [0] = (__sig = 1129270852, __opaque = char [40] @ 0x0000000166ab49c0) [1] = (__sig = 1129270852, __opaque = char [40] @ 0x0000000166ab49f0) } } } _class_to_be_initialized = nullptr _SleepEvent = 0x0000000134244d00 } The JavaThread's _jvmti_thread_state = 0x00000001336aafc0 value matches the current JvmtiThreadState so it looks like _thread_saved = 0x0000000135029410 is a valid JavaThread*. That JavaThread* value also appears on the current threads list: Threads class SMR info: _java_thread_list=0x0000000125558250, length=22, elements={ 0x0000000135808210, 0x000000013582b610, 0x000000013582e410, 0x000000013582f010, 0x000000012580ae10, 0x000000013582fc10, 0x000000012580ba10, 0x000000013500c410, 0x0000000126008210, 0x00000001348c5810, 0x00000001348d6010, 0x0000000133810a10, 0x0000000136011410, 0x0000000135028c10, 0x0000000135808e10, 0x000000012584d410, 0x0000000135830810, 0x0000000135029410, 0x0000000136011c10, 0x0000000135831010, 0x000000013582c210, 0x00000001338b8410 } so it is a valid and protected JavaThread*.
22-02-2023
Here's the logs from my jdk-21+10 stress run sighting on macosx-aarch64: $ unzip -l jdk-21+10_macosx-aarch64.8303086.zip Archive: jdk-21+10_macosx-aarch64.8303086.zip Length Date Time Name --------- ---------- ----- ---- 23717 02-18-2023 05:06 jdk-21+10_2/failures.macosx-aarch64/GetStackTraceSuspendedStressTest.jtr.slowdebug 111326 02-22-2023 14:04 jdk-21+10_2/failures.macosx-aarch64/hs_err_pid5235.log --------- ------- 135043 2 files
22-02-2023