Bug ID: JDK-8256572 assert(n->_left->_parent == n) failed: sanity in share/memory/metaspace/blockTree.cpp:113

JDK-8256572 : assert(n->_left->_parent == n) failed: sanity in share/memory/metaspace/blockTree.cpp:113

Type: Bug
Component: hotspot
Sub-Component: runtime
Affected Version: 16

Priority: P2
Status: Closed
Resolution: External

Submitted: 2020-11-18
Updated: 2021-01-26
Resolved: 2020-12-03

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 16
16Resolved

Related Reports

Relates :	JDK-8256371 - Compiler thread crashes in ClassLoaderData::is_alive()
Relates :	JDK-8256725 - Metaspace: better blocktree and binlist asserts

Description

Tests runtime/Metaspace/elastic/TestMetaspaceAllocationMT1.java#debug
runtime/Metaspace/elastic/TestMetaspaceAllocationMT2.java#debug
fail with 
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/c909db09-6c9b-402f-b3f3-aca8c6860a96/runs/991019b7-aa93-4076-a8df-1733e5f45dc3/workspace/open/src/hotspot/share/memory/metaspace/blockTree.cpp:113), pid=6047, tid=15700
#  assert(n->_left->_parent == n) failed: sanity
#
# JRE version: Java(TM) SE Runtime Environment (16.0+24) (fastdebug build 16-ea+24-1553)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 16-ea+24-1553, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x6aaa75]  metaspace::BlockTree::verify() const+0x2c5
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/scratch/1/core.6047)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Dtest.vm.opts=-XX:MaxRAMPercentage=6 -Djava.io.tmpdir=/opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/tmp -Dtest.tool.vm.opts=-J-XX:MaxRAMPercentage=6 -J-Djava.io.tmpdir=/opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/tmp -Dtest.compiler.opts= -Dtest.java.opts= -Dtest.jdk=/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/linux-x64-debug.jdk/jdk-16/fastdebug -Dcompile.jdk=/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/linux-x64-debug.jdk/jdk-16/fastdebug -Dtest.timeout.factor=4.0 -Dtest.nativepath=/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/linux-x64-debug.test/hotspot/jtreg/native -Dtest.root=/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/src.full/open/test/hotspot/jtreg -Dtest.name=runtime/Metaspace/elastic/TestMetaspaceAllocationMT2.java#debug -Dtest.file=/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/src.full/open/test/hotspot/jtreg/runtime/Metaspace/elastic/TestMetaspaceAllocationMT2.java -Dtest.src=/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/src.full/open/test/hotspot/jtreg/runtime/Metaspace/elastic -Dtest.src.path=/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/src.full/open/test/hotspot/jtreg/runtime/Metaspace/elastic:/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/src.full/open/test/lib -Dtest.classes=/opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/classes/3/runtime/Metaspace/elastic/TestMetaspaceAllocationMT2_debug.d -Dtest.class.path=/opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/classes/3/runtime/Metaspace/elastic/TestMetaspaceAllocationMT2_debug.d:/opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/classes/3/test/lib -Dtest.class.path.prefix=/opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/classes/3/runtime/Metaspace/elastic/TestMetaspaceAllocationMT2_debug.d:/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/src.full/open/test/hotspot/jtreg/runtime/Metaspace/elastic:/opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/classes/3/test/lib -Dtest.modules=java.base/jdk.internal.misc java.management --add-modules=java.base,java.management --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED -XX:MaxRAMPercentage=6 -Djava.io.tmpdir=/opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/tmp -Djava.library.path=/opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/linux-x64-debug.test/hotspot/jtreg/native -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -XX:VerifyMetaspaceInterval=10 com.sun.javatest.regtest.agent.MainWrapper /opt/mach5/mesos/work_dir/slaves/47535081-0322-4d83-bb78-f7e11abb86d0-S45878/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/48733db1-c40f-4ae0-a546-db94fc04b00a/runs/41147e4f-a88a-49c8-855a-e1dba02241c6/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/runtime/Metaspace/elastic/TestMetaspaceAllocationMT2_debug.d/main.0.jta

Host: ol7-build-test-961943, AMD EPYC 7551 32-Core Processor, 8 cores, 31G, Oracle Linux Server release 7.8
Time: Sat Nov 14 13:52:17 2020 UTC elapsed time: 32.181143 seconds (0d 0h 0m 32s)

---------------  T H R E A D  ---------------

Current thread (0x00007fb5ac066de0):  JavaThread "Thread-79" [_thread_in_vm, id=15700, stack(0x00007fb5f13f3000,0x00007fb5f14f4000)]

Stack: [0x00007fb5f13f3000,0x00007fb5f14f4000],  sp=0x00007fb5f14f2620,  free space=1021k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x6aaa75]  metaspace::BlockTree::verify() const+0x2c5
V  [libjvm.so+0x1447215]  metaspace::MetaspaceArena::verify_locked() const+0x65
V  [libjvm.so+0x14483b2]  metaspace::MetaspaceArena::allocate(unsigned long)+0x6d2
V  [libjvm.so+0x19cc596]  WB_AllocateFromMetaspaceTestArena+0x126
J 299  sun.hotspot.WhiteBox.allocateFromMetaspaceTestArena(JJ)J (0 bytes) @ 0x00007fb60d3dc7c1 [0x00007fb60d3dc6e0+0x00000000000000e1]
J 436 c2 RandomAllocator.allocateRandomly()V (57 bytes) @ 0x00007fb60d403514 [0x00007fb60d403140+0x00000000000003d4]
J 449 c1 RandomAllocator.tick()V (52 bytes) @ 0x00007fb605f2875c [0x00007fb605f286a0+0x00000000000000bc]
J 438 c2 RandomAllocatorThread.run()V (64 bytes) @ 0x00007fb60d4029a8 [0x00007fb60d402900+0x00000000000000a8]
v  ~StubRoutines::call_stub
V  [libjvm.so+0xe38955]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x595
V  [libjvm.so+0xe391d5]  JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x4c5
V  [libjvm.so+0xe3967c]  JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0xac
V  [libjvm.so+0xfccd5b]  thread_entry(JavaThread*, Thread*)+0x12b
V  [libjvm.so+0x1886f56]  JavaThread::thread_main_inner()+0x256
V  [libjvm.so+0x188dc10]  Thread::call_run()+0x100
V  [libjvm.so+0x1570066]  thread_native_entry(Thread*)+0x116

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 299  sun.hotspot.WhiteBox.allocateFromMetaspaceTestArena(JJ)J (0 bytes) @ 0x00007fb60d3dc74c [0x00007fb60d3dc6e0+0x000000000000006c]
J 436 c2 RandomAllocator.allocateRandomly()V (57 bytes) @ 0x00007fb60d403514 [0x00007fb60d403140+0x00000000000003d4]
J 449 c1 RandomAllocator.tick()V (52 bytes) @ 0x00007fb605f2875c [0x00007fb605f286a0+0x00000000000000bc]
J 438 c2 RandomAllocatorThread.run()V (64 bytes) @ 0x00007fb60d4029a8 [0x00007fb60d402900+0x00000000000000a8]
v  ~StubRoutines::call_stub
Register to memory mapping:

RAX=0x00007fb624928000 points into unknown readable memory: 0x0000000000000058 | 58 00 00 00 00 00 00 00
RBX=0x00007fb5f0984a00 points into unknown readable memory: 0x00007fb5f0eadd98 | 98 dd ea f0 b5 7f 00 00
RCX=0x00007fb623882920: <offset 0x0000000001c3c920> in /opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/linux-x64-debug.jdk/jdk-16/fastdebug/lib/server/libjvm.so at 0x00007fb621c46000
RDX=0x00007fb6236fef58: <offset 0x0000000001ab8f58> in /opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/linux-x64-debug.jdk/jdk-16/fastdebug/lib/server/libjvm.so at 0x00007fb621c46000
RSP=0x00007fb5f14f2620 is pointing into the stack for thread: 0x00007fb5ac066de0
RBP=0x00007fb5f14f2710 is pointing into the stack for thread: 0x00007fb5ac066de0
RSI=0x0000000000000071 is an unknown value
RDI=0x00007fb6236fec38: <offset 0x0000000001ab8c38> in /opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/linux-x64-debug.jdk/jdk-16/fastdebug/lib/server/libjvm.so at 0x00007fb621c46000
R8 =0x0 is NULL
R9 =0x00007fb5f14f1bf0 is pointing into the stack for thread: 0x00007fb5ac066de0
R10=0x000000000000001f is an unknown value
R11=0x0 is NULL
R12=0xffffffffffffffff is an unknown value
R13=0x0000000000000003 is an unknown value
R14=0x00007fb5a8000d78 points into unknown readable memory: 0x00007fb623c1a020 | 20 a0 c1 23 b6 7f 00 00
R15=0x0000000000000004 is an unknown value


Registers:
RAX=0x00007fb624928000, RBX=0x00007fb5f0984a00, RCX=0x00007fb623882920, RDX=0x00007fb6236fef58
RSP=0x00007fb5f14f2620, RBP=0x00007fb5f14f2710, RSI=0x0000000000000071, RDI=0x00007fb6236fec38
R8 =0x0000000000000000, R9 =0x00007fb5f14f1bf0, R10=0x000000000000001f, R11=0x0000000000000000
R12=0xffffffffffffffff, R13=0x0000000000000003, R14=0x00007fb5a8000d78, R15=0x0000000000000004
RIP=0x00007fb6222f0a75, EFLAGS=0x0000000000010287, CSGSFS=0x002b000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fb5f14f2620)
0x00007fb5f14f2620:   00007fb5b0017348 0000000000000000
0x00007fb5f14f2630:   0000000000000034 00007fb5f14f2650
0x00007fb5f14f2640:   000000000000004d 0000000000000159
0x00007fb5f14f2650:   00007fb623bd97b0 ffff804a0eb0d9af 

Instructions: (pc=0x00007fb6222f0a75)
0x00007fb6222f0975:   0f 11 46 10 8b 85 58 ff ff ff 48 c7 85 38 ff ff
0x00007fb6222f0985:   ff 00 00 00 00 85 c0 0f 8e c3 02 00 00 66 0f 1f
0x00007fb6222f0995:   44 00 00 83 e8 01 89 85 58 ff ff ff 48 98 48 c1
0x00007fb6222f09a5:   e0 05 48 03 85 60 ff ff ff 48 8b 48 10 48 8b 18
0x00007fb6222f09b5:   44 8b 68 08 48 8b 40 18 48 89 8d 30 ff ff ff 49
0x00007fb6222f09c5:   89 c4 41 81 fd 0f 27 00 00 0f 8f 1c 05 00 00 4c
0x00007fb6222f09d5:   8b 53 20 4d 85 d2 0f 85 61 05 00 00 48 8b 03 49
0x00007fb6222f09e5:   39 5e 08 0f 84 89 05 00 00 48 85 c0 0f 84 f9 07
0x00007fb6222f09f5:   00 00 48 8b 43 20 48 83 f8 04 0f 86 db 05 00 00
0x00007fb6222f0a05:   48 39 85 30 ff ff ff 0f 83 3e 06 00 00 49 39 c4
0x00007fb6222f0a15:   0f 86 9d 06 00 00 48 8b 43 08 48 85 c0 0f 84 aa
0x00007fb6222f0a25:   00 00 00 48 39 d8 0f 84 17 08 00 00 48 39 18 74
0x00007fb6222f0a35:   52 48 8d 05 db 5b 9e 01 4c 89 f7 48 8b 30 e8 d8
0x00007fb6222f0a45:   f5 ff ff 48 8b 43 08 48 39 18 74 37 48 8d 05 50
0x00007fb6222f0a55:   a7 93 01 48 8d 0d c1 1e 59 01 48 8d 15 f2 e4 40
0x00007fb6222f0a65:   01 be 71 00 00 00 48 8d 3d c6 e1 40 01 48 8b 00
0x00007fb6222f0a75:   c6 00 58 31 c0 e8 41 2b 3b 00 e8 0c bc eb 00 48
0x00007fb6222f0a85:   8b 43 08 48 8b 4b 20 48 63 b5 58 ff ff ff 41 8d
0x00007fb6222f0a95:   55 01 3b b5 5c ff ff ff 0f 84 6d 06 00 00 8d 7e
0x00007fb6222f0aa5:   01 48 c1 e6 05 48 03 b5 60 ff ff ff f3 0f 7e 85
0x00007fb6222f0ab5:   30 ff ff ff 89 bd 58 ff ff ff 66 48 0f 6e c9 48
0x00007fb6222f0ac5:   89 06 66 0f 6c c1 89 56 08 0f 11 46 10 48 8b 43
0x00007fb6222f0ad5:   10 48 85 c0 0f 84 a4 00 00 00 48 39 d8 0f 84 a8
0x00007fb6222f0ae5:   06 00 00 48 39 18 74 52 48 8d 05 24 5b 9e 01 4c
0x00007fb6222f0af5:   89 f7 48 8b 30 e8 21 f5 ff ff 48 8b 43 10 48 39
0x00007fb6222f0b05:   18 74 37 48 8d 05 99 a6 93 01 48 8d 0d 0a 1e 59
0x00007fb6222f0b15:   01 48 8d 15 63 e4 40 01 be 7d 00 00 00 48 8d 3d
0x00007fb6222f0b25:   0f e1 40 01 48 8b 00 c6 00 58 31 c0 e8 8a 2a 3b
0x00007fb6222f0b35:   00 e8 55 bb eb 00 48 8b 43 10 f3 0f 7e 43 20 48
0x00007fb6222f0b45:   63 b5 58 ff ff ff 41 83 c5 01 3b b5 5c ff ff ff
0x00007fb6222f0b55:   0f 84 f5 05 00 00 8d 56 01 48 c1 e6 05 48 03 b5
0x00007fb6222f0b65:   60 ff ff ff 89 95 58 ff ff ff 66 49 0f 6e d4 48 


Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007fb5b0017348 points into unknown readable memory: 0x00007fb5f0b1a400 | 00 a4 b1 f0 b5 7f 00 00
stack at sp + 1 slots: 0x0 is NULL
stack at sp + 2 slots: 0x0000000000000034 is an unknown value
stack at sp + 3 slots: 0x00007fb5f14f2650 is pointing into the stack for thread: 0x00007fb5ac066de0
stack at sp + 4 slots: 0x000000000000004d is an unknown value
stack at sp + 5 slots: 0x0000000000000159 is an unknown value
stack at sp + 6 slots: 0x00007fb623bd97b0: <offset 0x0000000001f937b0> in /opt/mach5/mesos/work_dir/jib-master/install/jdk-16+24-1553/linux-x64-debug.jdk/jdk-16/fastdebug/lib/server/libjvm.so at 0x00007fb621c46000
stack at sp + 7 slots: 0xffff804a0eb0d9af is an unknown value

Comments

Both JDK-8256371 and this item are assumed to be caused by machine problems.
07-12-2020
Thats a relief, thanks. For reference, Ioi asked me to look at our tests, and we found no occurrence of either this or JDK-8256371 (we keep a database of hs-err files).
04-12-2020
[~iklam] concurs. Closing as External.
03-12-2020
This failure only happened once and it happened on the same machine that also had JDK-8256371, which is a suspected test machine problem. I'm looking at the core file and it doesn't look like it should assert here. 185 for (Block* b = _blocks[i]; b != NULL; b = b->_next) { 186 assert(b->_word_size == s, "bad block size"); <===== assert here 187 local_counter.add(s); 188 } In the debugger it's perfectly fine: (gdb) print *b $3 = {_next = 0x7f3e2ae5bfe8, _word_size = 3} (gdb) print s $4 = 3 (gdb) print i $5 = 1 (gdb) print MinWordSize $6 = 2 I don't think this is a bug in our code so it should be closed as "External" like the other bug.
03-12-2020
JDK-8256725: "Metaspace: better blocktree and binlist asserts" was pushed to help with analysis. Since I cannot reproduce it anywhere on our machines, please add any new occurrences (both the jtr files and the hs-err files) to this ticket. My current theory atm is that this is a metaspace overwriter unrelated to the asserting code, since both breakages are in very different unrelated data structures. JDK-8256725 adds canaries and more tests.
25-11-2020
ILW = HLH = P2
24-11-2020
Never happened in any of our systems. Not reproducable. I created a PR to beef up assertion and error printouts: https://github.com/openjdk/jdk/pull/1339
20-11-2020
Is this intermittent or reproducable? Are there other occurrences? Anything special about this setup? Would it be possible to get the other hs-err file? (since these seem to be two different asserts)
20-11-2020
May I have the jtr file as well? That should contain debug output about the block tree.
20-11-2020