JDK-8246727 : nsk/jvmti/scenarios/capability/CM03/cm03t001/TestDescription.java goes for rare JVM crash
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 11,14,15
  • Priority: P4
  • Status: Resolved
  • Resolution: Cannot Reproduce
  • Submitted: 2020-06-08
  • Updated: 2022-03-23
  • Resolved: 2022-03-23
Related Reports
Relates :  
JDK-8246381 - VM crashes with "Current BasicObjectLock* below than low_mark"
Relates :  
JDK-8254874 - ZGC: JNIHandleBlock verification failure in stack watermark processing
Description
Testcase : nsk/jvmti/scenarios/capability/CM03/cm03t001/TestDescription.java
OS specific: NO.
Regression : Can't say (crash is very rare)
Special VM flag : Shows the failure with -XX:-TieredCompilation 
Small Log :

# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/interpreterRuntime.cpp:1077
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/workspace/open/src/hotspot/share/interpreter/interpreterRuntime.cpp:1077), pid=12217, tid=12282
#  assert(!(((ThreadShadow*)__the_thread__)->has_pending_exception())) failed: Should not have any exceptions pending
#
# JRE version: Java(TM) SE Runtime Environment (14.0.2+6) (fastdebug build 14.0.2+6-38)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 14.0.2+6-38, mixed mode, sharing, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xc94466]ThreadState(thread='MainThread', state='debuggeeWaiting') waiting for state debuggeeThreadDeath
  InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+0x186
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/805146e6-8fdb-4552-bf9e-385b73cf7129-S328/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2e4579ee-a17c-4f5e-93b8-b6d583b3905e/runs/eef28afb-b23b-48f6-99d2-81f0297a17fe/testoutput/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti/scratch/2/core.12217)
#
# An error report file with more information is saved as:
# /805146e6-8fdb-4552-bf9e-385b73cf7129-S328/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2e4579ee-a17c-4f5e-93b8-b6d583b3905e/runs/eef28afb-b23b-48f6-99d2-81f0297a17fe/testoutput/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti/scratch/2/hs_err_pid12217.log
#
Comments
Restoring the "cannot reproduce" resolution for this old bug. I do not have time to investigate it more deeply and the code has changed too much since.
23-03-2022
Got this same error on a JDK11u build running on Linux + G1GC.
12-03-2021
I think that [~stefank]'s work on this bug is relevant here: JDK-8254874 ZGC: JNIHandleBlock verification failure in stack watermark processing
16-10-2020
I am reopening this issue to investigate how it was that an exception came to be pending when we executed the counter overflow check (and this triggered the assertion failure). The logic that made the async exception pending should have initiated the throwing of the exception rather than continuing "forward".
22-07-2020
Yes the fix for JDK-8246381 avoids the assertion failure reported here. It is still of interest to us to know exactly where the async exception was installed in this JVM TI test case. So I'll be looking at this testcase further.
21-07-2020
This issue is no more reproducible with the fix of JDK-8246381. In JDK-8246381 issue async exception is handled in frequency_counter_overflow_inner. Which avoid current issue to occur further down in the same method. They are not really duplicate but affected with async exceptions. Closing as cannot reproduce any more.
20-07-2020
This bug will be present in older releases, but may be much harder to trigger due to the changes in safepoints/handshakes that alters the likelihood of which async exception check will install the async exception. The async exception is being installed, but not processed, when we do the backward branch in the loop and then go into the runtime in response to the frequency counter overflow - that code doesn't expect to be called with an exception pending. We need to ensure we notice the pending exception before calling that code. But as per JDK-8246381 there is also an issue if the async exception is installed by Java code that executes as a result of calling frequency_counter_overflow.
23-06-2020
-Xlog:exceptions=debug gives more details about the async exception ThreadState(thread='MainThread', state='init') waiting for state debuggeeStarted ThreadState(thread='Debuggee Thread', state='init') set state to debuggeeStarted ThreadState(thread='Debuggee Thread', state='debuggeeStarted') waiting for state jvmtiInited ThreadState(thread='MainThread', state='debuggeeStarted') got state debuggeeStarted ThreadState(thread='MainThread', state='debuggeeStarted') set state to jvmtiInited ThreadState(thread='MainThread', state='jvmtiInited') waiting for state debuggeeWaiting ThreadState(thread='Debuggee Thread', state='jvmtiInited') got state jvmtiInited ThreadState(thread='Debuggee Thread', state='jvmtiInited') set state to debuggeeWaiting ThreadState(thread='MainThread', state='debuggeeWaiting') got state debuggeeWaiting [0.298s][info][exceptions] Pending Async. exception installed of type: java.lang.ThreadDeath [0.298s][info][exceptions] Async. exception installed at runtime exit (0x00007fd5ec034ff0) (pc: 0x00007fd630d44993 sp: 0x00007fd62c343860 ) of type: java.lang.ThreadDeath # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/interpreterRuntime.cpp:1077 # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (/opt/mach5/mesos/work_dir/slaves/4728e7c1-7e67-490e-be0f-6bbf2a2f33db-S77/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/0bb335a6-5ceb-4943-b056-3382f27e7179/runs/95e9e54d-95fa-4840-9008-523f1cec6a98/workspace/open/src/hotspot/share/interpreter/interpreterRuntime.cpp:1077), pid=20751, tid=20772 # assert(!(((ThreadShadow*)__the_thread__)->has_pending_exception())) failed: Should not have any exceptions pending # # JRE version: Java(TM) SE Runtime Environment (16.0) (fastdebug build 16-internal+0-2020-06-22-0933124.fairoz.matte.jdk) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 16-internal+0-2020-06-22-0933124.fairoz.matte.jdk, mixed mode, sharing, compressed oops, g1 gc, linux-amd64) # Problematic frame: # V [libjvm.so+0xce7b53]ThreadState(thread='MainThread', state='debuggeeWaiting') waiting for state debuggeeThreadDeath InterpreterRuntime::frequency_counter_overflow_inner(JavaThread*, unsigned char*)+0x153 #
22-06-2020
See discussions in JDK-8246381.
22-06-2020
We used to perform async exception checks on backward branches (as part of safepoint/handshake polls). If we detected the ThreadDeath before we look at the overflow counter we will just abort the loop by throwing the exception. But now we don't check on the backward branch so the loop just keeps running and we hit the counteroverflow. That code can lead to Java code executing (per the comments in that count). If that Java code involved a call to a method (seems exceedingly likely - how else do we execute Java!) then the check in the JavaCallWrapper will install the async exception. The Java code we were trying to invoke would abort with the exception and we get back to the IRT code which then checks for pending exceptions and is surprised to find one. Just a theory though. The devil is in the detail of this code. JDK-8246381 is also an issue that may be caused by a change in the async exception delivery policy.
19-06-2020