Bug ID: JDK-8235696 ZGC: assert(ZNMethod::is_armed(nm) == _should_disarm

JDK-8235696 : ZGC: assert(ZNMethod::is_armed(nm) == _should_disarm_nmethods) failed: Invalid state

Type: Bug
Component: hotspot
Sub-Component: gc
Affected Version: 14

Priority: P2
Status: Closed
Resolution: Fixed

Submitted: 2019-12-10
Updated: 2020-01-31
Resolved: 2019-12-12

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 14	JDK 15
14 b27Fixed	15Fixed

Related Reports

Relates :	JDK-8173361 - various crashes in JvmtiExport::post_compiled_method_load
Relates :	JDK-8235273 - nmethodLocker not needed for COMPILED_METHOD_UNLOAD events
Relates :	JDK-8234654 - ZGC: Only disarm NMethods when marking/relocating code roots
Relates :	JDK-8212160 - JVMTI agent crashes with "assert(_value != 0LL) failed: resolving NULL _value"

Description

Kitchensink fails with 
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (open/src/hotspot/share/gc/z/zRootsIterator.cpp:156), pid=2052, tid=2057
#  assert(ZNMethod::is_armed(nm) == _should_disarm_nmethods) failed: Invalid state
#
# JRE version: Java(TM) SE Runtime Environment (14.0+27) (fastdebug build 14-ea+27-1297)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 14-ea+27-1297, mixed mode, tiered, z gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x1830d23]  ZRootsIteratorCodeBlobClosure::do_code_blob(CodeBlob*)+0x73
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/fcf4c0c4-d73e-4321-860c-6613427db92b-S1793/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/8436a87f-0cb7-458a-ada1-a7ec3f525fe6/runs/2d8ffc73-0050-4316-a8b7-ad7fa3cc3930/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/core.2052)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -XX:MaxRAMPercentage=6 -XX:+CreateCoredumpOnCrash -XX:+UnlockExperimentalVMOptions -XX:+UseZGC -XX:MaxRAMPercentage=50 -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:+CrashOnOutOfMemoryError -Djava.net.preferIPv6Addresses=false -XX:+DisplayVMOutputToStderr -Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags -XX:+DisableExplicitGC -XX:+StartAttachListener --add-exports=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED -Djava.io.tmpdir=/opt/mach5/mesos/work_dir/slaves/fcf4c0c4-d73e-4321-860c-6613427db92b-S1793/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/8436a87f-0cb7-458a-ada1-a7ec3f525fe6/runs/2d8ffc73-0050-4316-a8b7-ad7fa3cc3930/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/java.io.tmpdir -Duser.home=/opt/mach5/mesos/work_dir/slaves/fcf4c0c4-d73e-4321-860c-6613427db92b-S1793/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/8436a87f-0cb7-458a-ada1-a7ec3f525fe6/runs/2d8ffc73-0050-4316-a8b7-ad7fa3cc3930/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/user.home -agentpath:/opt/mach5/mesos/work_dir/jib-master/install/jdk-14+27-1297/linux-x64-debug.test/hotspot/jtreg/native/libJvmtiStressModule.so -XX:NativeMemoryTracking=detail applications.kitchensink.process.stress.Main /opt/mach5/mesos/work_dir/slaves/fcf4c0c4-d73e-4321-860c-6613427db92b-S1793/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/8436a87f-0cb7-458a-ada1-a7ec3f525fe6/runs/2d8ffc73-0050-4316-a8b7-ad7fa3cc3930/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink_java/scratch/0/kitchensink.final.properties

Host: Intel(R) Xeon(R) Platinum 8167M CPU @ 2.00GHz, 8 cores, 58G, Oracle Linux Server release 7.6
Time: Tue Dec 10 16:35:24 2019 UTC elapsed time: 633 seconds (0d 0h 10m 33s)

---------------  T H R E A D  ---------------

Current thread (0x00007f550c08b780):  GCTaskThread "ZWorker#0" [stack: 0x00007f5511884000,0x00007f5511984000] [id=2057]

Stack: [0x00007f5511884000,0x00007f5511984000],  sp=0x00007f5511982c00,  free space=1019k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1830d23]  ZRootsIteratorCodeBlobClosure::do_code_blob(CodeBlob*)+0x73
V  [libjvm.so+0x15200a8]  ServiceThread::oops_do(OopClosure*, CodeBlobClosure*)+0x38
V  [libjvm.so+0x1830daf]  ZRootsIteratorThreadClosure::do_thread(Thread*)+0x5f
V  [libjvm.so+0x16990c9]  Threads::possibly_parallel_threads_do(bool, ThreadClosure*)+0x169
V  [libjvm.so+0x182eb08]  ZRootsIterator::oops_do(ZRootsIteratorClosure*)+0x1f8
V  [libjvm.so+0x18066f0]  ZMarkRootsTask::work()+0x20
V  [libjvm.so+0x183a09d]  ZTask::GangTask::work(unsigned int)+0x1d
V  [libjvm.so+0x17c4cd4]  GangWorker::run_task(WorkData)+0x84
V  [libjvm.so+0x17c4e18]  GangWorker::loop()+0x48
V  [libjvm.so+0x169bbb6]  Thread::call_run()+0xf6
V  [libjvm.so+0x13c9cae]  thread_native_entry(Thread*)+0x10e

Register to memory mapping:

RAX=0x00007f5515434000 points into unknown readable memory: 58 00 00 00 00 00 00 00
RBX=0x00007f54fc93c010 is at entry_point+-496 in (nmethod*)0x00007f54fc93c010
RCX=0x00007f551436cc10: <offset 0x00000000019cbc10> in /opt/mach5/mesos/work_dir/jib-master/install/jdk-14+27-1297/linux-x64-debug.jdk/jdk-14/fastdebug/lib/server/libjvm.so at 0x00007f55129a1000
RDX=0x00007f551434b058: <offset 0x00000000019aa058> in /opt/mach5/mesos/work_dir/jib-master/install/jdk-14+27-1297/linux-x64-debug.jdk/jdk-14/fastdebug/lib/server/libjvm.so at 0x00007f55129a1000
RSP=0x00007f5511982c00 points into unknown readable memory: 50 2c 98 11 55 7f 00 00
RBP=0x00007f5511982c10 points into unknown readable memory: 40 2c 98 11 55 7f 00 00
RSI=0x000000000000009c is an unknown value
RDI=0x00007f551434b028: <offset 0x00000000019aa028> in /opt/mach5/mesos/work_dir/jib-master/install/jdk-14+27-1297/linux-x64-debug.jdk/jdk-14/fastdebug/lib/server/libjvm.so at 0x00007f55129a1000
R8 =0x000043ab02000004 is an unknown value
R9 =0x0000000000000001 is an unknown value
R10=0x0 is NULL
R11=0x00007f52df000000 points into unknown readable memory: 0f 00 00 00 00 00 00 00
R12=0x00007f5511982c50 points into unknown readable memory: f0 61 70 14 55 7f 00 00
R13=0x0000000000000240 is an unknown value
R14=0x00007f5298192468 points into unknown readable memory: 00 00 00 00 00 00 00 00
R15=0x00007f5511982ce0 points into unknown readable memory: 18 62 70 14 55 7f 00 00


Registers:
RAX=0x00007f5515434000, RBX=0x00007f54fc93c010, RCX=0x00007f551436cc10, RDX=0x00007f551434b058
RSP=0x00007f5511982c00, RBP=0x00007f5511982c10, RSI=0x000000000000009c, RDI=0x00007f551434b028
R8 =0x000043ab02000004, R9 =0x0000000000000001, R10=0x0000000000000000, R11=0x00007f52df000000
R12=0x00007f5511982c50, R13=0x0000000000000240, R14=0x00007f5298192468, R15=0x00007f5511982ce0
RIP=0x00007f55141d1d23, EFLAGS=0x0000000000010297, CSGSFS=0x002b000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f5511982c00)
0x00007f5511982c00:   00007f5511982c50 00007f54f41ad6f8
0x00007f5511982c10:   00007f5511982c40 00007f5513ec10a8
0x00007f5511982c20:   00007f5511982ce0 00007f550c5777d0
0x00007f5511982c30:   00007f5511982ce0 0000000000000240 

Instructions: (pc=0x00007f55141d1d23)
0x00007f55141d1c23:   07 ff 50 10 48 8b 03 48 85 c0 74 09 5b 41 5c 5d
0x00007f55141d1c33:   c3 0f 1f 40 00 5b 49 83 44 24 10 01 41 5c 5d c3
0x00007f55141d1c43:   66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 8d 05
0x00007f55141d1c53:   a1 a6 53 00 55 be 8c 01 00 00 48 8d 3d c4 93 17
0x00007f55141d1c63:   00 48 8b 00 48 89 e5 c6 00 58 e8 7e 02 1d ff 5d
0x00007f55141d1c73:   e9 b8 09 b9 ff 0f 1f 84 00 00 00 00 00 48 8d 05
0x00007f55141d1c83:   71 a6 53 00 55 be 8c 01 00 00 48 8d 3d 94 93 17
0x00007f55141d1c93:   00 48 8b 00 48 89 e5 c6 00 58 e8 4e 02 1d ff 5d
0x00007f55141d1ca3:   e9 88 09 b9 ff 0f 1f 84 00 00 00 00 00 55 48 89
0x00007f55141d1cb3:   e5 41 54 49 89 fc 48 89 f7 53 48 8b 06 48 89 f3
0x00007f55141d1cc3:   ff 50 10 48 85 db 74 10 84 c0 74 0c 48 89 df e8
0x00007f55141d1cd3:   59 6a b1 ff 84 c0 75 05 5b 41 5c 5d c3 49 8b 74
0x00007f55141d1ce3:   24 08 48 89 df e8 73 04 fe ff 48 89 df e8 0b 04
0x00007f55141d1cf3:   fe ff 41 0f b6 54 24 10 38 d0 74 39 48 8d 05 f2
0x00007f55141d1d03:   a5 53 00 48 8d 15 4b 93 17 00 48 8d 0d fc ae 19
0x00007f55141d1d13:   00 be 9c 00 00 00 48 8d 3d 08 93 17 00 48 8b 00
0x00007f55141d1d23:   c6 00 58 31 c0 e8 13 fd 1c ff e8 fe 08 b9 ff 41
0x00007f55141d1d33:   0f b6 54 24 10 84 d2 74 9f 48 89 df 5b 41 5c 5d
0x00007f55141d1d43:   e9 e8 03 fe ff 0f 1f 84 00 00 00 00 00 55 48 8d
0x00007f55141d1d53:   05 98 44 53 00 48 8d 0d 61 e8 f9 ff 48 89 e5 41
0x00007f55141d1d63:   54 49 89 fc 53 48 89 f3 48 83 ec 20 48 8b 77 08
0x00007f55141d1d73:   48 89 45 d0 48 89 75 d8 48 8b 06 48 8b 50 28 31
0x00007f55141d1d83:   c0 48 39 ca 75 47 48 8d 0d 5c bd 53 00 88 45 e0
0x00007f55141d1d93:   48 8d 55 d0 48 89 df 48 8b 03 80 39 00 b9 00 00
0x00007f55141d1da3:   00 00 48 0f 44 d1 ff 90 c8 00 00 00 49 8b 7c 24
0x00007f55141d1db3:   08 48 8d 15 f5 e7 f9 ff 48 8b 07 48 8b 40 20 48
0x00007f55141d1dc3:   39 d0 75 19 48 83 c4 20 5b 41 5c 5d c3 48 89 f7
0x00007f55141d1dd3:   ff d2 49 8b 74 24 08 eb ad 0f 1f 40 00 48 89 de
0x00007f55141d1de3:   ff d0 48 83 c4 20 5b 41 5c 5d c3 66 90 0f 1f 80
0x00007f55141d1df3:   00 00 00 00 00 00 00 00 00 00 00 00 00 48 8d 05
0x00007f55141d1e03:   d5 0e 5e 00 8b 00 c3 66 0f 1f 44 00 00 48 89 f8
0x00007f55141d1e13:   c3 66 66 2e 0f 1f 84 00 00 00 00 00 90 e9 4b 33 


Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007f5511982c50 points into unknown readable memory: f0 61 70 14 55 7f 00 00
stack at sp + 1 slots: 0x00007f54f41ad6f8 points into unknown readable memory: 58 5a 70 14 55 7f 00 00
stack at sp + 2 slots: 0x00007f5511982c40 points into unknown readable memory: 80 2c 98 11 55 7f 00 00
stack at sp + 3 slots: 0x00007f5513ec10a8: <offset 0x00000000015200a8> in /opt/mach5/mesos/work_dir/jib-master/install/jdk-14+27-1297/linux-x64-debug.jdk/jdk-14/fastdebug/lib/server/libjvm.so at 0x00007f55129a1000
stack at sp + 4 slots: 0x00007f5511982ce0 points into unknown readable memory: 18 62 70 14 55 7f 00 00
stack at sp + 5 slots: 0x00007f550c5777d0 is a thread
stack at sp + 6 slots: 0x00007f5511982ce0 points into unknown readable memory: 18 62 70 14 55 7f 00 00
stack at sp + 7 slots: 0x0000000000000240 is an unknown value

Comments

KS doesn't fail
31-01-2020
URL: https://hg.openjdk.java.net/jdk/jdk/rev/9468dada3479 User: pliden Date: 2019-12-12 09:39:37 +0000
12-12-2019
Correction to my above comment. I assumed we took the first branch when walking the code cache and posting events for all nmethods: if (state == NULL) { ServiceThread::enqueue_deferred_event(&event); } else { state->enqueue_event(&event); } but that code takes the second branch. I've managed to reproduce the same assert but taking that path instead with: make -C ../build/fastdebug test-only TEST=test/hotspot/jtreg/serviceability/jvmti/CompiledMethodLoad/Zombie.java JTREG="RETAIN=all;VERBOSE=all;JAVA_OPTIONS=-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xmx8m -Xlog:gc" and: while true; do jcmd com.sun.javatest.regtest.agent.MainWrapper GC.run; done So, the actual failure mode in the original assert is probably different, but likely similar.
10-12-2019
JDK-8173361 is the patch that adds nmethods/oops_do to the deferred event queue in the ServiceThread, where the assertion is being fired. JDK-8212160 adds the test that fails.
10-12-2019
The following is a hint that this has to do with JDK-8212160: V [libjvm.so+0x15200a8] ServiceThread::oops_do(OopClosure, CodeBlobClosure)+0x38
10-12-2019
JDK-8234654 added the assert, and it is correct. The cause of this is likely to be found in: JDK-8212160: JVMTI agent crashes with "assert(_value != 0LL) failed: resolving NULL _value" The assumption was that nmethods that we post events for are newly created, therefore newly registered, therefore disarmed. However, this is not true when we bulk generate events for all live nmethods. Then there's no guarantee that the posted nmethods are disarmed.
10-12-2019