JDK-8212160 : JVMTI agent crashes with "assert(_value != 0LL) failed: resolving NULL _value"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jvmti
  • Affected Version: 11,12
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2018-10-13
  • Updated: 2023-11-21
  • Resolved: 2019-12-05
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 13 JDK 14 JDK 8
11.0.10-oracleFixed 13.0.6Fixed 14 b27Fixed 8u411Fixed
Related Reports
Duplicate :  
JDK-8212159 - assert(zombie_ok || !nm->is_zombie()) failed: cannot lock a zombie method
Relates :  
JDK-8181110 - jvmti/hotswap test crashes in Method::checked_resolve_jmethod_id(_jmethodID*)
Relates :  
JDK-8236772 - Fix build for windows 32-bit after 8212160 and 8234331.
Relates :  
JDK-8235456 - Minimal VM is broken after JDK-8212160
Relates :  
JDK-8235273 - nmethodLocker not needed for COMPILED_METHOD_UNLOAD events
Relates :  
JDK-8236124 - Minimal VM slowdebug build failed after JDK-8212160
Relates :  
JDK-8235696 - ZGC: assert(ZNMethod::is_armed(nm) == _should_disarm_nmethods) failed: Invalid state
Description
While debugging JDK-8212159 with slowdebug JVM variant I hit one more crash:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (c:\Users\Andrei\java\jdk\src\hotspot\share\runtime/handles.hpp:168), pid=7160, tid=804
#  assert(_value != 0LL) failed: resolving NULL _value
#
# JRE version: OpenJDK Runtime Environment (11.0+28) (slowdebug build 11+28)
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 12-internal+0-adhoc.Andrei.jdk, compiled mode, tiered, compressed oops, g1 gc, windows-amd64)
# Core dump will be written. Default location: C:\Users\Andrei\java\jvmti-race\hs_err_pid7160.mdmp
#

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0xf333e1]  VMError::report_and_die+0x81  (vmerror.cpp:1269)
V  [jvm.dll+0x5c51ae]  report_vm_error+0x8e  (debug.cpp:232)
V  [jvm.dll+0x1f7bc]  methodHandle::non_null_obj+0x4c  (handles.hpp:168)
V  [jvm.dll+0x1e443]  methodHandle::operator->+0x13  (handles.hpp:168)
V  [jvm.dll+0xad68f8]  JvmtiEventMark::to_jmethodID+0x18  (jvmtiexport.cpp:199)
V  [jvm.dll+0xad57b0]  JvmtiMethodEventMark::JvmtiMethodEventMark+0x40  (jvmtiexport.cpp:235)
V  [jvm.dll+0xad53d6]  JvmtiCompiledMethodLoadEventMark::JvmtiCompiledMethodLoadEventMark+0x46  (jvmtiexport.cpp:1062)
V  [jvm.dll+0xad1752]  JvmtiExport::post_compiled_method_load+0x232  (jvmtiexport.cpp:2162)
V  [jvm.dll+0xa52a80]  JvmtiCodeBlobEvents::generate_compiled_method_load_events+0x90  (jvmticodeblobevents.cpp:241)
V  [jvm.dll+0xabef70]  JvmtiEnv::GenerateEvents+0x50  (jvmtienv.cpp:3552)
V  [jvm.dll+0xa6c231]  jvmti_GenerateEvents+0x1e1  (jvmtienter.cpp:5420)
C  [jvmtiCompiledZombie.dll+0x1047]

It can be easily reproduced with slowdebug build with the following command:
java -agentpath:jvmtiCompiledZombie.dll -Xcomp -XX:ReservedCodeCacheSize=20m -XX:+ZombieALot -jar SwingSet2.jar

The agent source is attached.
Comments
Fix request (13u) Requesting backport to 13u for parity with 11u. The patch doesn't apply cleanly due to minor context differences in nmethod.cpp and serviceThread.cpp (JDK-8235273 and JDK-8170299 are not in 13u), reapplied manually. Tested with tier1 and other jvmti tests. Follow-up fixes JDK-8235456, JDK-8236124 and relevant part of JDK-8236772 are planned to be backported as well. RFR: http://mail.openjdk.java.net/pipermail/jdk-updates-dev/2021-January/004731.html
25-01-2021
Hi Christoph, I already fixed the Windows build in the current webrev as else it would not have passed our builds. I would leave the mentioned change in state "not downported" as the other issue fixed there is still open.
23-11-2020
And JDK-8236772 is another follow up which should be taken as well, corect?
20-11-2020
Fix request (11u) -- will label after testing completed. I would like to downport this for parity with 11.0.10-oracle. I had to resolve and adapt the change: http://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-November/004088.html There are also two follow ups for Minimal VM I'll push together.
20-11-2020
[~pli] That issue has been fixed by Jie Fu under JDK-8235456
06-12-2019
Hi, Minimal VM build is broken after this patch pushed. src/hotspot/share/prims/jvmtiImpl.hpp:523:46: error: return-statement with a value, in function returning 'void' [-fpermissive] void post(JvmtiEnv* env) NOT_JVMTI_RETURN_(false); It looks like the newly added void method shouldn't have a return value. To reproduce, use "bash configure --with-jvm-variants=minimal && make".
06-12-2019
URL: https://hg.openjdk.java.net/jdk/jdk/rev/366c0f357ee6 User: coleenp Date: 2019-12-05 21:58:50 +0000
05-12-2019
Update: The idea above to add these events to the ServiceThread deferred event queue can cause a native OOM error with the test case that failed. That's because the JavaThread posting the events from the CodeCache in a loop can keep adding events on the queue and the ServiceThread cannot keep up. The fix that works better is to have a instance of DeferredEvents on JavaThread for the thread that posts the events from the code cache. The fix I have out for review simply reuses the DeferredEventQueue code to post the events after the CodeCache_lock is dropped.
02-12-2019
The race seems to be that for the late attach: jvmtiError JvmtiCodeBlobEvents::generate_compiled_method_load_events(JvmtiEnv* env) walks the code cache with the CodeCache_lock set. When it drops the CodeCache_lock to process a method, the sweeper thread might be sweeping the method concurrently. The sweeper thread also drops the CodeCache_lock so setting the nmethodLocker inside the CodeCache_lock in generate_compiled_method_load_events does no good. I added the lock_count to the sweeper LogSweeper logging code, and these are the records for the nmethod that fails: $4 = 299 $5 = {traversal = 1394, compile_id = 1545, traversal_mark = 1392, state = 4, kind = 0x0, vep = 0x7fbc8f33dac0, uep = 0x7fbc8f33dac0 , locked = 1, line = 703} $6 = 298 $7 = {traversal = 1394, compile_id = 1545, traversal_mark = 1392, state = 2, kind = 0x0, vep = 0x7fbc8f33dac0, uep = 0x7fbc8f33dac0, locked = 0, line = 676} So the sweeper observed the nmethod as unlocked at the beginning of the function (read upward) but the logging after it turned it into zombie, now finds it locked. I think we should add these compiled method load events to the deferred queue and keep them alive (from unloading and zombie) using the mechanism I hope to add soon for JDK-8173361.
20-11-2019
I can reproduce this crash with the test case from bug JDK-8212159. I think it's the same thing. The nmethod is unloaded when we safepoint while getting the jmethodIDs. jmethodID takes out a lock that safepoint checks: JmethodIdCreation_lock Based on the short lifetime of this locked region, it should not safepoint check, and a NoSafepointVerifier should be added to generate_compiled_method_load_events or post_compiled_method_load. In the crash, the nmethod that we're doing a compiled method load with is unloaded. _orig_pc_offset = 48, _compile_id = 8212, _comp_level = 4, _has_flushed_dependencies = true, _unload_reported = true, _state = 4 '\004', _oops_are_stale = true, _rtm_state = NoRTM, _lock_count = 1, _stack_traversal_mark = 5400, State is zombie.
19-11-2019
Interesting issue. Will investigate it when there is a chance.
29-10-2019