JDK-8225064 : [Graal] Application SEGV in G1ParScanThreadState::copy_to_survivor_space(G1HeapRegionAttr, oopDesc*, markOopDesc*)+0x48
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 13
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • Submitted: 2019-05-30
  • Updated: 2020-06-30
  • Resolved: 2019-06-25
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 13 JDK 14
13Fixed 14 b05Resolved
Related Reports
Duplicate :  
Relates :  
Relates :  
Description
Application running with Graal crashes with 
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007efd5ffedd08, pid=29784, tid=29809
#
# JRE version: Java(TM) SE Runtime Environment (13.0) (build 13-internal+0-2019-05-30-0201567.leonid.mesnik.ks-apps)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (13-internal+0-2019-05-30-0201567.leonid.mesnik.ks-apps, mixed mode, sharing, tiered, jvmci, jvmci compiler, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x761d08]  G1ParScanThreadState::copy_to_survivor_space(G1HeapRegionAttr, oopDesc*, markOopDesc*)+0x48
#
# Core dump will be written. Default location: /scratch/opt/mach5/mesos/work_dir/slaves/df27b84d-b5c1-4760-9e48-df95fd33274c-S251/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/9b7f059b-085c-4c19-b146-b4508fd8374e/runs/39307470-19db-4fc6-bc44-76a2571501b5/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_runthese_RunThese24H_java/scratch/0/core.29784
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -XX:MaxRAMPercentage=6 -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI -XX:+TieredCompilation -XX:+UseJVMCICompiler -Djvmci.Compiler=graal -ea -esa --add-opens=java.base/java.net=ALL-UNNAMED -Dseed=2666051056755231 -XX:MaxRAMPercentage=50 applications.runthese.Runner -duration 1440 -runlist /scratch/opt/mach5/mesos/work_dir/slaves/df27b84d-b5c1-4760-9e48-df95fd33274c-S251/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/9b7f059b-085c-4c19-b146-b4508fd8374e/runs/39307470-19db-4fc6-bc44-76a2571501b5/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_runthese_RunThese24H_java/scratch/0/./RunTheseTestList.dat

Host: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, 8 cores, 59G, Oracle Linux Server release 7.1
Time: Thu May 30 11:30:42 2019 UTC elapsed time: 33086 seconds (0d 9h 11m 26s)

---------------  T H R E A D  ---------------

Current thread (0x00007efcdc002800):  GCTaskThread "GC Thread#2" [stack: 0x00007efcf4b81000,0x00007efcf4c81000] [id=29809]

Stack: [0x00007efcf4b81000,0x00007efcf4c81000],  sp=0x00007efcf4c7fb30,  free space=1018k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x761d08]  G1ParScanThreadState::copy_to_survivor_space(G1HeapRegionAttr, oopDesc*, markOopDesc*)+0x48
V  [libjvm.so+0x7629bb]  G1ParScanThreadState::trim_queue()+0x40b
V  [libjvm.so+0x71daa2]  G1ParEvacuateFollowersClosure::do_void()+0x52
V  [libjvm.so+0x724482]  G1EvacuateRegionsTask::evacuate_live_objects(G1ParScanThreadState*, unsigned int)+0x82
V  [libjvm.so+0x71fcde]  G1EvacuateRegionsBaseTask::work(unsigned int)+0xae
V  [libjvm.so+0xdf7f7d]  GangWorker::loop()+0x4d
V  [libjvm.so+0xd63e3d]  Thread::call_run()+0x10d
V  [libjvm.so+0xbb5597]  thread_native_entry(Thread*)+0xe7


siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000ba17b6bb8

Register to memory mapping:

RAX=0x0000000000000001 is an unknown value
RBX=0x0000000000000101 is an unknown value
RCX=0x0000000000000003 is an unknown value
RDX=0x00007efd60a25230: <offset 0x0000000001199230> in /scratch/opt/mach5/mesos/work_dir/jib-master/install/2019-05-30-0201567.leonid.mesnik.ks-apps/linux-x64.jdk/jdk-13/lib/server/libjvm.so at 0x00007efd5f88c000
RSP=0x00007efcf4c7fb30 points into unknown readable memory: 60 05 04 58 fd 7e 00 00
RBP=0x00007efcf4c7fba0 points into unknown readable memory: 10 fc c7 f4 fc 7e 00 00
RSI=0x0000000000000101 is an unknown value
RDI=0x0000000ba17b6bb0 is an unknown value
R8 =0x0 is NULL
R9 =0x0000000000000001 is an unknown value
R10=0x0000000000000002 is an unknown value
R11=0x0000000000000001 is an unknown value
R12=0x00007efc980ae840 points into unknown readable memory: 48 8d 96 60 fd 7e 00 00
R13=0x00000000eac3cb80 is pointing into object: [B 
{0x00000000eac3c098} - klass: {type array byte}
 - length: 7168
R14=0x2f62696c2f6b636a is an unknown value
R15=0x00007efc980ae840 points into unknown readable memory: 48 8d 96 60 fd 7e 00 00


Registers:
RAX=0x0000000000000001, RBX=0x0000000000000101, RCX=0x0000000000000003, RDX=0x00007efd60a25230
RSP=0x00007efcf4c7fb30, RBP=0x00007efcf4c7fba0, RSI=0x0000000000000101, RDI=0x0000000ba17b6bb0
R8 =0x0000000000000000, R9 =0x0000000000000001, R10=0x0000000000000002, R11=0x0000000000000001
R12=0x00007efc980ae840, R13=0x00000000eac3cb80, R14=0x2f62696c2f6b636a, R15=0x00007efc980ae840
RIP=0x00007efd5ffedd08, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007efcf4c7fb30)
0x00007efcf4c7fb30:   00007efd58040560 00007efd58061310
0x00007efcf4c7fb40:   0000000100000000 0000000000000001
0x00007efcf4c7fb50:   0000000000000002 00007efcf4c70001
0x00007efcf4c7fb60:   00007efcf4c7fd38 00007efd60439c01 

Instructions: (pc=0x00007efd5ffedd08)
0x00007efd5ffedc08:   3c 01 00 00 4d 8b 7e 50 4c 89 4d c0 48 89 4d c8
0x00007efd5ffedc18:   e8 53 44 08 00 4c 8b 4d c0 48 8b 4d c8 4d 89 f8
0x00007efd5ffedc28:   48 89 c2 48 8d 35 7a 41 70 00 48 8d 3d bf 18 74
0x00007efd5ffedc38:   00 31 c0 e8 90 15 fa ff 49 8b 7d 08 41 8b b5 98
0x00007efd5ffedc48:   01 00 00 4c 89 e1 48 89 da e8 0a 41 fb ff 41 8b
0x00007efd5ffedc58:   86 3c 01 00 00 49 8d bd 70 01 00 00 d1 e8 83 e0
0x00007efd5ffedc68:   01 41 89 85 90 01 00 00 48 8d 05 35 e1 a9 00 80
0x00007efd5ffedc78:   38 00 74 3c 48 8d 05 ad 75 a3 00 8b 53 08 8b 48
0x00007efd5ffedc88:   08 48 d3 e2 48 03 10 48 63 4a 0c 48 8d 05 26 d1
0x00007efd5ffedc98:   a9 00 48 89 de ff 14 c8 48 83 c4 18 48 89 d8 5b
0x00007efd5ffedca8:   41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 44 00 00
0x00007efd5ffedcb8:   48 8b 53 08 eb d1 66 90 55 48 89 e5 41 57 41 56
0x00007efd5ffedcc8:   49 89 ce 41 55 49 89 d5 41 54 49 89 fc 53 89 f3
0x00007efd5ffedcd8:   0f be c7 48 83 ec 48 66 89 45 b8 48 8d 05 c2 e0
0x00007efd5ffedce8:   a9 00 0f b6 00 84 c0 0f 84 bb 03 00 00 8b 7a 08
0x00007efd5ffedcf8:   48 8d 15 31 75 a3 00 8b 4a 08 48 d3 e7 48 03 3a
0x00007efd5ffedd08:   8b 4f 08 85 c9 0f 8e 05 03 00 00 f6 c1 01 0f 84
0x00007efd5ffedd18:   64 01 00 00 48 8b 07 4c 89 ee ff 90 00 01 00 00
0x00007efd5ffedd28:   89 45 b0 49 8b 44 24 08 4c 89 ea 4c 63 55 b0 c7
0x00007efd5ffedd38:   45 a8 00 00 00 00 48 8b 80 f8 00 00 00 8b 48 68
0x00007efd5ffedd48:   48 8b 40 58 48 d3 ea 80 7d b8 01 48 8b 04 d0 8b
0x00007efd5ffedd58:   80 7c 01 00 00 89 45 a4 0f 84 fa 02 00 00 48 0f
0x00007efd5ffedd68:   be 45 b8 45 31 db 41 80 bc 24 c8 01 00 00 00 49
0x00007efd5ffedd78:   8d 84 44 60 01 00 00 44 0f b6 78 01 44 8a 18 74
0x00007efd5ffedd88:   0a 41 80 ff 02 0f 84 54 03 00 00 49 8b 44 24 58
0x00007efd5ffedd98:   49 0f be d7 48 8b 8c d0 10 01 00 00 8b 90 28 01
0x00007efd5ffedda8:   00 00 48 89 c7 85 d2 74 0a 41 80 ff 01 0f 84 85
0x00007efd5ffeddb8:   03 00 00 48 8b 59 30 48 8b 41 38 48 29 d8 48 c1
0x00007efd5ffeddc8:   e8 03 49 39 c2 0f 87 75 04 00 00 4a 8d 04 d3 48
0x00007efd5ffeddd8:   89 41 30 48 85 db 0f 84 5c 04 00 00 48 8d 05 fd
0x00007efd5ffedde8:   80 a3 00 48 8b 00 0f 18 0c 03 48 89 da 4c 89 f0
0x00007efd5ffeddf8:   48 83 ca 03 f0 49 0f b1 55 00 49 39 c6 74 59 48 


Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007efd58040560 points into unknown readable memory: 08 00 00 00 00 00 00 00
stack at sp + 1 slots: 0x00007efd58061310 points into unknown readable memory: 90 13 06 58 fd 7e 00 00
stack at sp + 2 slots: 0x0000000100000000 is an unallocated location in the heap
stack at sp + 3 slots: 0x0000000000000001 is an unknown value
stack at sp + 4 slots: 0x0000000000000002 is an unknown value
stack at sp + 5 slots: 0x00007efcf4c70001 points into unknown readable memory: 00 00 00 00 00 00 00
stack at sp + 6 slots: 0x00007efcf4c7fd38 points into unknown readable memory: 00 00 00 00 00 00 00 00
stack at sp + 7 slots: 0x00007efd60439c01: <offset 0x0000000000badc01> in /scratch/opt/mach5/mesos/work_dir/jib-master/install/2019-05-30-0201567.leonid.mesnik.ks-apps/linux-x64.jdk/jdk-13/lib/server/libjvm.so at 0x00007efd5f88c000

Comments
Note: While hgupdater and the mercurial history shows that this change went into jdk/jdk as part of the bulk integration, the change was manually reverted as part of the corresponding merge changeset (http://hg.openjdk.java.net/jdk/jdk/rev/ea3b1a8fd4bb), meaning the end result (after the merge changeset) is that the change is *not* actually in jdk/jdk.
09-07-2019

JDK-8225497 was pushed into JDK 14.
28-06-2019

Hi Tobias, ok, i will wait till fix get in, just to verify the final fix. as per Tom, it is not pushed yet.
20-06-2019

Next Graal Update will be JDK-8225497.
20-06-2019

as all iteration passed, and original issue is tracked by labs, will be closing the bug as duplicate to next graal update!
20-06-2019

i think this is the issue Tom is referring, two allocation happening back to back. and second objects initialization store the first reference and then first objects initialization store the second without a barrier. There is an instance[1] of this in my crash (for synthetic final case). [1] 0x00007f03874ca022: mov 0x140(%r15),%rsi 0x00007f03874ca029: lea 0x20(%rsi),%r10 0x00007f03874ca02d: movabs $0x1002635f8,%r11 0x00007f03874ca037: cmp 0x150(%r15),%r10 0x00007f03874ca03e: ja 0x00007f03874cabdc 0x00007f03874ca044: mov %r10,0x140(%r15) 0x00007f03874ca04b: prefetchnta 0xe0(%rsi) 0x00007f03874ca052: mov 0xb8(%r11),%r10 0x00007f03874ca059: mov %r10,(%rsi) 0x00007f03874ca05c: movl $0x2004c6bf,0x8(%rsi) 0x00007f03874ca063: movl $0x0,0xc(%rsi) 0x00007f03874ca06a: movq $0x0,0x10(%rsi) 0x00007f03874ca072: movq $0x0,0x18(%rsi) 0x00007f03874ca07a: movl $0xffffffff,0xc(%rsi) 0x00007f03874ca081: mov 0x140(%r15),%r10 0x00007f03874ca088: lea 0x30(%r10),%r8 0x00007f03874ca08c: movabs $0x100262bd8,%rcx 0x00007f03874ca096: cmp 0x150(%r15),%r8 0x00007f03874ca09d: ja 0x00007f03874caba1 0x00007f03874ca0a3: mov %r8,0x140(%r15) 0x00007f03874ca0aa: prefetchnta 0xf0(%r10) 0x00007f03874ca0b2: mov 0xb8(%rcx),%r8 0x00007f03874ca0b9: mov %r8,(%r10) 0x00007f03874ca0bc: movl $0x2004c57b,0x8(%r10) 0x00007f03874ca0c4: movl $0x0,0xc(%r10) 0x00007f03874ca0cc: movq $0x0,0x10(%r10) 0x00007f03874ca0d4: movq $0x0,0x18(%r10) 0x00007f03874ca0dc: movq $0x0,0x20(%r10) 0x00007f03874ca0e4: movq $0x0,0x28(%r10) 0x00007f03874ca0ec: mov %r11,0x58(%rsp) 0x00007f03874ca0f1: mov %rsi,%r8 0x00007f03874ca0f4: mov %r8d,0x28(%r10) 0x00007f03874ca0f8: mov %rdi,%rcx 0x00007f03874ca0fb: mov %ecx,0x20(%r10) 0x00007f03874ca0ff: mov %r14,%rdx 0x00007f03874ca102: mov %edx,0x1c(%r10) 0x00007f03874ca106: mov %ebx,0x18(%r10) 0x00007f03874ca10a: movb $0x1,0x16(%r10) 0x00007f03874ca10f: mov %r10,%r8 0x00007f03874ca112: mov %r8d,0x14(%rsi) 0x00007f03874ca116: mov %r8d,0x18(%rsi) 0x00007f03874ca11a: mov %r10,0x50(%rsp) 0x00007f03874ca11f: callq 0x00007f0387484360
20-06-2019

i have a done couple of runs already no more crashes!!, i used to get crashes in every run before!!
19-06-2019

it is highly likely it is the same issue.. i will run some iterations and confirm, Thank you so much!!
19-06-2019

I think this is a recently diagnosed issue with mishandling of ReduceInitialCardMarks in our CommitAllocation. if you run with -XX:-ReduceInitialCardMarks and the problem goes away then it's the same.
19-06-2019

verifying if issue exist in jdk8. and if it is some data structure changes triggering the issue.!
19-06-2019

previous comment deleted as it was YG allocation, and doesn;t require a barrier!
19-06-2019

some cases it is StringBuilder [B field, some case it is synthetic final field.
19-06-2019

Hi Igor, there are missing barriers, VerifyRememberedSets points to it. Thank you!!
19-06-2019

[830.771s][error][gc,verify] GC(206) ---------- [830.771s][error][gc,verify] GC(206) Missing rem set entry: [830.771s][error][gc,verify] GC(206) Field 0x00000000f9fb09ec of obj 0x00000000f9fb09d8 in region 159:(O)[0x00000000f9f00000,0x00000000f9fffff8,0x00000000fa000000] [830.771s][error][gc,verify] GC(206) java.lang.StringBuilder [830.771s][error][gc,verify] GC(206) {0x00000000f9fb09d8} - klass: 'java/lang/StringBuilder' [830.771s][error][gc,verify] GC(206) - ---- fields (total size 3 words): [830.771s][error][gc,verify] GC(206) - 'count' 'I' @12 6 [830.771s][error][gc,verify] GC(206) - 'coder' 'B' @16 0 [830.771s][error][gc,verify] GC(206) - 'value' '[B' @20 [B{0x00000000fff00000} (fff00000) [830.771s][error][gc,verify] GC(206) points to obj 0x00000000fff00000 in region 255:(E)[0x00000000fff00000,0x0000000100000000,0x0000000100000000] remset Complete [830.771s][error][gc,verify] GC(206) [B [830.771s][error][gc,verify] GC(206) {0x00000000fff00000} - klass: {type array byte} [830.771s][error][gc,verify] GC(206) - length: 16 [830.771s][error][gc,verify] GC(206) - 0: 32 2 [830.771s][error][gc,verify] GC(206) - 1: 33 3 [830.771s][error][gc,verify] GC(206) - 2: 32 2 [830.771s][error][gc,verify] GC(206) - 3: 2e . [830.771s][error][gc,verify] GC(206) - 4: 35 5 [830.771s][error][gc,verify] GC(206) - 5: 35 5 [830.771s][error][gc,verify] GC(206) - 6: 0 [830.771s][error][gc,verify] GC(206) - 7: 0 [830.771s][error][gc,verify] GC(206) - 8: 0 [830.771s][error][gc,verify] GC(206) - 9: 0 [830.771s][error][gc,verify] GC(206) - 10: 0 [830.771s][error][gc,verify] GC(206) - 11: 0 [830.771s][error][gc,verify] GC(206) - 12: 0 [830.771s][error][gc,verify] GC(206) - 13: 0 [830.771s][error][gc,verify] GC(206) - 14: 0 [830.771s][error][gc,verify] GC(206) - 15: 0 [830.771s][error][gc,verify] GC(206) Obj head CTE = -1, field CTE = -1. [830.771s][error][gc,verify] GC(206) ---------- To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/heapRegion.cpp:825 # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (open/src/hotspot/share/gc/g1/heapRegion.cpp:825), pid=31258, tid=31265 # guarantee(!failures) failed: HeapRegion RemSet verification failed # # JRE version: Java(TM) SE Runtime Environment (13.0) (fastdebug build 13-internal+0-2019-06-13-0623350.jamsheed.null) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 13-internal+0-2019-06-13-0623350.jamsheed.null, mixed mode, tiered, jvmci, jvmci compiler, compressed oops, g1 gc, linux-amd64) # Problematic frame: # V [libjvm.so+0xd442f4] HeapRegion::verify_rem_set() const+0x44 # # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /mnt/jamsheed/work/OpenGrok/source/jdk_tip/jdk/specjbb/core.31258) # # An error report file with more information is saved as: # /mnt/jamsheed/work/OpenGrok/source/jdk_tip/jdk/specjbb/hs_err_pid31258.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # Current thread is 31265 Dumping core ... Aborted (core dumped) Wed Jun 19 07:09:38 IST 2019
19-06-2019

Hi Igor, Yet to successfully reproduce the scenario with verify*GC options, had tried VerifyRememberedSets(alone) initially when i started my analysis, but was not useful at that point(i.e it was crashing without more information). i need to verify this again!(speaking from memory, and need to confirm again) bad oop that i shared was from core file analysis.
19-06-2019

Try running with VerifyRememberedSets, it's not automatically on by Verify.*GC by default. That should help catch the missing barrier problems.
18-06-2019

bad oop is most of the time byte[] field in StringBuilder.
18-06-2019

i tried everything in my laptop, will try Verify*GC in mach5 m/cs.
18-06-2019

Hi Vladimir, with Verify*GC i couldn't reproduce the issue(may be as runs are too slow). Issue is reproducible with verifyOops, but it never detected the anomaly(may be as oops are all good at inserted barriers). it always manifested in crash with an oop pointing some already moved regions.
18-06-2019

[~jcm] Try to run with -XX:+VerifyBeforeGC -XX:+VerifyAfterGC. May be do separate run with -XX:+VerifyOops.
18-06-2019

or if it is like g1 barrier itself is not generated properly.
17-06-2019

oops point to already moved locations, could it be like copy gc happened between write and a barrier ?
14-06-2019

specjbb2005 with 2m new gen crashes everytime.
13-06-2019

issue is easily reproducible. investigating
13-06-2019

reopened as JDK-8223796 is not an issue as per analysis!
12-06-2019

Thanks for analysis Tobias, yes i see classes not live in core files, could be same issue, closing as duplicate for now.
11-06-2019

will save core files.
11-06-2019

This is likely related to JDK-8223794.
07-06-2019