JDK-8302351 : "assert(!JavaThread::current()->is_interp_only_mode() || !nm->method()->is_continuation_enter_intrinsic() || ContinuationEntry::is_interpreted_call(return_pc)) failed: interp_only_mode but not in enterSpecial interpreted entry" in fixup_callers_callsite
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 21
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux,os_x
  • CPU: x86_64,aarch64
  • Submitted: 2023-02-13
  • Updated: 2023-07-25
  • Resolved: 2023-07-06
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 21 JDK 22
21Fixed 22 b06Fixed
Related Reports
Relates :  
JDK-8310340 - assert(_thread->is_interp_only_mode() || stub_caller) failed: expected a stub-caller
Relates :  
JDK-8288949 - serviceability/jvmti/vthread/ContStackDepthTest/ContStackDepthTest.java failing
Relates :  
JDK-8218403 - Deoptimizing top frame not always sufficient to reach interpreter-only mode
Relates :  
JDK-8311516 - JavaThread can continue executing in compiled code after switch to interpreter only mode
Relates :  
JDK-8304354 - nsk/jdi/Event/equals/equals001 fails: assert(!state->is_interp_only_mode()) failed: sanity check
Description
The following test failed in the JDK21 CI:

applications/kitchensink/Kitchensink.java

Here's a snippet from the log file:

[2023-02-11T20:43:15.889002068Z] Gathering output for process 2310136
[2023-02-11T20:43:22.081309540Z] Waiting for completion for process 2310136
[2023-02-11T20:43:22.081390780Z] Waiting for completion finished for process 2310136
Output and diagnostic info for process 2310136 was saved into 'pid-2310136-output.log'
[stress.process.out] #
[stress.process.out] # A fatal error has been detected by the Java Runtime Environment:
[stress.process.out] #
[stress.process.out] #  Internal Error (/opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S151618/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/a4d65992-3afb-47b6-b336-7e34fa3a803f/runs/750ecb11-b813-450b-8537-da6a29c4cd6a/workspace/open/src/hotspot/share/runtime/sharedRuntime.cpp:2132), pid=2301175, tid=2310378
[stress.process.out] #  assert(!JavaThread::current()->is_interp_only_mode() || !nm->method()->is_continuation_enter_intrinsic() || ContinuationEntry::is_interpreted_call(return_pc)) failed: interp_only_mode but not in enterSpecial interpreted entry
[stress.process.out] #
[stress.process.out] # JRE version: Java(TM) SE Runtime Environment (21.0+10) (fastdebug build 21-ea+10-LTS-728)
[stress.process.out] # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 21-ea+10-LTS-728, compiled mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
[stress.process.out] # Problematic frame:
[stress.process.out] # V  [libjvm.so+0x16a9bf0]  SharedRuntime::fixup_callers_callsite(Method*, unsigned char*)+0x2e0
[stress.process.out] #
[stress.process.out] # Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/91e16c40-06d4-468a-9fc3-7198a5bb7d5a-S33376/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6d2b36a6-871e-4ef2-9072-52d82df296fd/runs/9e712550-718c-400e-afa6-b22ae22c70f7/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/core.2301175)
[stress.process.out] #
[stress.process.out] Unsupported internal testing APIs have been used.
[stress.process.out] 
[stress.process.out] # An error report file with more information is saved as:
[stress.process.out] # /opt/mach5/mesos/work_dir/slaves/91e16c40-06d4-468a-9fc3-7198a5bb7d5a-S33376/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6d2b36a6-871e-4ef2-9072-52d82df296fd/runs/9e712550-718c-400e-afa6-b22ae22c70f7/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/hs_err_pid2301175.log
[stress.process.out] #
[stress.process.out] # If you would like to submit a bug report, please visit:
[stress.process.out] #   https://bugreport.java.com/bugreport/crash.jsp
[stress.process.out] #
[2023-02-11T20:44:22.086497406Z] Gathering output for process 2310520
[2023-02-11T20:44:22.094622277Z] Waiting for completion for process 2310520
[2023-02-11T20:44:22.094691037Z] Waiting for completion finished for process 2310520


Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x0000fffe800dc2a0):  JavaThread "ForkJoinPool-1-worker-33" daemon [_thread_in_Java, id=2310378, stack(0x0000fffea17a0000,0x0000fffea19a0000)]

Stack: [0x0000fffea17a0000,0x0000fffea19a0000],  sp=0x0000fffea199dbf0,  free space=2038k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x16a9bf0]  SharedRuntime::fixup_callers_callsite(Method*, unsigned char*)+0x2e0  (sharedRuntime.cpp:2132)
v  ~BufferBlob::I2C/C2I adapters 0x0000ffff843ff190
J 16823  jdk.internal.vm.Continuation.enterSpecial(Ljdk/internal/vm/Continuation;ZZ)V java.base@21-ea (0 bytes) @ 0x0000ffff85c12ed8 [0x0000ffff85c12dc0+0x0000000000000118]

[error occurred during error reporting (printing native stack (with source info)), id 0xb, SIGSEGV (0xb) at pc=0x0000ffff9a74662c]

Retrying call stack printing without source information...
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x16a9bf0]  SharedRuntime::fixup_callers_callsite(Method*, unsigned char*)+0x2e0  (sharedRuntime.cpp:2132)
v  ~BufferBlob::I2C/C2I adapters 0x0000ffff843ff190
J 16823  jdk.internal.vm.Continuation.enterSpecial(Ljdk/internal/vm/Continuation;ZZ)V java.base@21-ea (0 bytes) @ 0x0000ffff85c12ed8 [0x0000ffff85c12dc0+0x0000000000000118]

[error occurred during error reporting (retry printing native stack (no source info)), id 0xb, SIGSEGV (0xb) at pc=0x0000ffff9a74662c]
Comments
A pull request was submitted for review. URL: https://git.openjdk.org/jdk21/pull/106 Date: 2023-07-10 14:23:18 +0000
10-07-2023
Changeset: 0c86c31b Author: Patricio Chilano Mateo <pchilanomate@openjdk.org> Date: 2023-07-06 19:15:45 +0000 URL: https://git.openjdk.org/jdk/commit/0c86c31bccd676e1cfbd35898ee16e89d5752688
06-07-2023
A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/14108 Date: 2023-05-23 22:07:37 +0000
23-05-2023
Thanks [~dlong]. I will finish testing the proposed fix then and send a PR to fix this crash. We can keep JDK-8218403 open if you think there are still other issues remaining with JVMCI.
22-05-2023
[~pchilanomate], it's been a long time since I thought about these problems, so my answers might not be 100% right. I think your proposed solution above would probably work for most cases. If I remember correctly, the hard cases for JDK-8218403 and when ForceEarlyReturn/PopFrame are involved, especially in the past when -Xcomp with JVMCI could cause a Java upcall in the resolve stub. But JVMCI no longer does that, so some of the problems that caused me to file JDK-8218403 may be impossible to reproduce now.
22-05-2023
[~dlong] I read the possible solutions listed in 8218403. Whats wrong with just checking if the thread is in interpreted only mode before returning to Java and in that case return the c2i adapter instead?: return current->is_interp_only_mode() ? callee_method->get_c2i_entry() : callee_method->verified_code_entry();
19-05-2023
The assert we hit was added along the changes for JDK-8288949 but this issue is pre-existent. It's just that now the assert made the bug visible. I see this is actually a duplicate of JDK-8218403. In JDK-8288949 there were changes made to resolve_static_call_C() that also deal with interpreted only mode. But the issue that was addressed in that bug was slightly different. It was trying to fix the case where the JavaThread already noticed it was in interpreted only mode but because enterSpecial didn't had an interpreted version we switched from interpreted to compiled and from there we incorrectly called the compiled version of enter() leading to asserts in JVMTI code about stack depth.
19-05-2023
Ok, after looking at all the wrong places in the code(JVMTI) I finally found the issue. I attached a reproducer. The issue is in all the calls to the runtime to resolve compiled callsites. An EnterInterpOnlyModeClosure handshake can caught the JavaThread at the JRT_BLOCK_END transition on those calls. None of those calls consider this scenario and just return callee_method->verified_code_entry() even if now the JavaThread is in interpreted only mode, e.g.: // resolve virtual call and update inline cache to monomorphic JRT_BLOCK_ENTRY(address, SharedRuntime::resolve_virtual_call_C(JavaThread* current)) methodHandle callee_method; JRT_BLOCK callee_method = SharedRuntime::resolve_helper(true, false, CHECK_NULL); current->set_vm_result_2(callee_method()); JRT_BLOCK_END // return compiled code entry point after potential safepoints assert(callee_method->verified_code_entry() != nullptr, " Jump to zero!"); return callee_method->verified_code_entry(); JRT_END The JavaThread will return back to the runtime stub where we restore all registers and will jump to callee_method->verified_code_entry(), which will be the compiled version of callee if there was one. The JavaThread will continue calling the compiled versions of methods without noticing it is in interpreted only mode until it either calls a method that hasn't been compiled yet or it returns to the caller of that resolved callee where the change to interpreter only mode happened (since the EnterInterpOnlyModeClosure handshake marked all the frames on the stack for deoptimization). So in our crashing case: A carrier is on its path to mount some virtual thread. Somewhere along that path, before executing notifyJvmtiMount() where JVMTI operations will be disabled for this thread until the transition finishes, the carrier calls one of this methods to resolve some callsite. An EnterInterpOnlyModeClosure happens on the JRT_BLOCK_END transition. On return to Java we still call the compiled version of that method and continue in compiled mode all the way up to enterSpecial. In enterSpecial the static callsite to enter() happens to go through the c2i adapter where we see there is a compiled version of enter() so we call fixup_callers_callsite() and hit the assert.
19-05-2023
This does not look like a JIT issue, moved to hotspot/runtime.
03-05-2023
V [libjvm.so+0x16a9bf0] SharedRuntime::fixup_callers_callsite(Method*, unsigned char*)+0x2e0 (sharedRuntime.cpp:2132) v ~BufferBlob::I2C/C2I adapters 0x0000ffff843ff190 J 16823 jdk.internal.vm.Continuation.enterSpecial(Ljdk/internal/vm/Continuation;ZZ)V java.base@21-ea (0 bytes) @ 0x0000ffff85c12ed8 [0x0000ffff85c12dc0+0x0000000000000118] It seems deoptimization happens when we process fixup request. There is no compiled method on top of call stack. There is comment about that in SharedRuntime::fixup_callers_callsite(): https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/sharedRuntime.cpp#L2038
22-03-2023
[~dlong] Hmm, I'm not sure, but a race with interp_only_mode sounds like a reasonable explanation.
14-02-2023
ILW = assert in debug build; seen in closed stress test only, intermittent; no workaround = MMH = P3
14-02-2023
[~rpressler], this assert was added by JDK-8288949. Could it be too strong, or maybe there is a race setting interp_only_mode?
14-02-2023