JDK-8273940 : vmTestbase/vm/mlvm/meth/stress/gc/callSequencesDuringGC/Test.java crashes in full gc during VM exit
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 18
  • Priority: P1
  • Status: Closed
  • Resolution: Fixed
  • Submitted: 2021-09-17
  • Updated: 2022-02-10
  • Resolved: 2021-09-20
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 18
18 b16Fixed
Related Reports
Relates :  
JDK-8274007 - [REDO] VM Exit does not abort concurrent mark
Relates :  
JDK-8273605 - VM Exit does not abort concurrent mark
Description
vmTestbase/vm/mlvm/meth/stress/gc/callSequencesDuringGC/Test.java crashes after passing the test during a full gc:

### TRACE 1: RNG seed = -3538409234787716714 (0xcee50b8c5a843196)
For random generator using seed: -3538409234787716714
To re-run test with same seed value please add "-Djdk.test.lib.random.seed=-3538409234787716714" to command line.
### TRACE 1: TEST PASSED
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=\\gc/g1/g1FullGCOopClosures.inline.hpp:91
[thread 47912 also had an error]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (...\\open\\src\\hotspot\\share\\gc/g1/g1FullGCOopClosures.inline.hpp:91), pid=55268, tid=50632
#  assert(G1CollectedHeap::heap()->is_in_reserved(forwardee)) failed: should be in object space
#
# JRE version: Java(TM) SE Runtime Environment (18.0+16) (fastdebug build 18-ea+16-845)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 18-ea+16-845, compiled mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64)
# Core dump will be written. Default location: ...\\test-support\\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\\scratch\\2\\hs_err_pid55268.mdmp
#
# An error report file with more information is saved as:
# T...\\jtreg_open_test_hotspot_jtreg_vmTestbase_vm_mlvm\\scratch\\2\\hs_err_pid55268.log
[thread 27724 also had an error]
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
Comments
Changeset: 4b3a4fff Author: Thomas Schatzl <tschatzl@openjdk.org> Date: 2021-09-20 16:17:52 +0000 URL: https://git.openjdk.java.net/jdk/commit/4b3a4fff39c1fba0d7eae719525e2a46b0a6d6ed
20-09-2021
I do not think this has much to do about archive regions after all. Initially I though maybe this is about aborting Concurrent Undo operations as most (all?) OSX crashes seem to be that way, but the Windows crash above disproves that theory. It looks like there is an issue with JDK-8273605. At least with that reverted, 2000 executions of this test are fine, while otherwise there is a ~2% failure rate.
20-09-2021
The # assert(G1CollectedHeap::heap()->is_in_reserved(forwardee)) failed: should be in object space error gives ~this stack trace (OSX) Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) crash: V [libjvm.dylib+0x5d649d] report_vm_error(char const*, int, char const*, char const*, ...)+0xdd V [libjvm.dylib+0x730e64] void G1AdjustClosure::adjust_pointer<oop>(oop*)+0x3c4 V [libjvm.dylib+0x5382db] ClassLoaderData::oops_do(OopClosure*, int, bool)+0x8b V [libjvm.dylib+0x53dc36] ClassLoaderDataGraph::roots_cld_do(CLDClosure*, CLDClosure*)+0x56 V [libjvm.dylib+0x797d74] G1RootProcessor::process_java_roots(G1RootClosures*, G1GCPhaseTimes*, unsigned int)+0xc4 V [libjvm.dylib+0x7982d6] G1RootProcessor::process_all_roots(OopClosure*, CLDClosure*, CodeBlobClosure*)+0x36 V [libjvm.dylib+0x72fba0] G1FullGCAdjustTask::work(unsigned int)+0x1a0 V [libjvm.dylib+0x1158d6e] GangWorker::run_task(WorkData)+0x5e Apparently bad oop in the CLDG? No particular observations with archive regions: | 130|0x00000007c8200000, 0x00000007c8200000, 0x00000007c8300000| 0%| F| |TAMS 0x00000007c8200000, 0x00000007c8200000| Untracked |1022|0x00000007ffe00000, 0x00000007ffe74000, 0x00000007fff00000| 45%|OA| |TAMS 0x00000007ffe74000, 0x00000007ffe74000| Untracked |1023|0x00000007fff00000, 0x00000007fff72000, 0x0000000800000000| 44%|CA| |TAMS 0x00000007fff00000, 0x00000007fff72000| Untracked Log output (added -Xlog:gc,gc+marking): [0.964s][info][gc,marking] GC(12) Concurrent Mark Abort [0.964s][info][gc ] GC(12) Concurrent Undo Cycle 61.103ms ### TRACE 1: TEST PASSED [1.038s][info][gc ] GC(14) Pause Young (Concurrent Start) (G1 Humongous Allocation) 4M->1M(10M) 42.141ms [1.038s][info][gc ] GC(15) Concurrent Undo Cycle [1.038s][info][gc,marking] GC(15) Concurrent Cleanup for Next Mark # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/g1FullGCOopClosures.inline.hpp:91 |1364|0x00000007ffe00000, 0x00000007ffe74000, 0x00000007fff00000| 45%|OA| |TAMS 0x00000007ffe74000, 0x00000007ffe74000| Untracked |1365|0x00000007fff00000, 0x00000007fff72000, 0x0000000800000000| 44%|CA| |TAMS 0x00000007fff00000, 0x00000007fff72000| Untracked
20-09-2021
Another # assert(!is_free(obj)) failed: Should not have objects in free regions. crash: 0.754s][info][gc ] GC(10) Concurrent Undo Cycle 1.351ms [0.898s][info][gc ] GC(11) Pause Full (System.gc()) 10M->1M(17M) 111.757ms ### TRACE 1: TEST PASSED [1.050s][info][gc ] GC(12) Pause Young (Concurrent Start) (G1 Humongous Allocation) 6M->4M(17M) 78.479ms [1.050s][info][gc ] GC(13) Concurrent Mark Cycle [1.050s][info][gc,marking] GC(13) Concurrent Clear Claimed Marks [1.050s][info][gc,marking] GC(13) Concurrent Clear Claimed Marks 0.007ms [1.050s][info][gc,marking] GC(13) Concurrent Scan Root Regions [1.050s][info][gc,marking] GC(13) Concurrent Scan Root Regions 0.187ms [1.050s][info][gc,marking] GC(13) Concurrent Mark [1.050s][info][gc,marking] GC(13) Concurrent Mark From Roots # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/g1FullGCHeapRegionAttr.hpp:69 # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (open/src/hotspot/share/gc/g1/g1FullGCHeapRegionAttr.hpp:69), pid=51351, tid=11779 # assert(!is_free(obj)) failed: Should not have objects in free regions. with the following stack trace: V [jvm.dll+0x4cda67] report_vm_error+0xb7 (debug.cpp:282) V [jvm.dll+0x5d0c8a] G1AdjustClosure::adjust_pointer<oopDesc *>+0x20a (g1FullGCOopClosures.inline.hpp:91) V [jvm.dll+0x451c98] ClassLoaderData::ChunkedHandleList::oops_do_chunk+0x58 (classLoaderData.cpp:212) V [jvm.dll+0x451b3c] ClassLoaderData::ChunkedHandleList::oops_do+0x3c (classLoaderData.cpp:224) V [jvm.dll+0x455379] ClassLoaderDataGraph::roots_cld_do+0x59 (classLoaderDataGraph.cpp:275) V [jvm.dll+0x6080fa] G1RootProcessor::process_java_roots+0xda (g1RootProcessor.cpp:193) V [jvm.dll+0x607fc3] G1RootProcessor::process_all_roots+0x43 (g1RootProcessor.cpp:140) V [jvm.dll+0x5d388a] G1FullGCAdjustTask::work+0x25a (g1FullGCAdjustTask.cpp:113) V [jvm.dll+0xd41eea] GangWorker::loop+0x8a (workgroup.cpp:239) That one does not have archive regions at all (on Windows?)
20-09-2021
Stack trace from one of the failures with the # assert(!is_free(obj)) failed: Should not have objects in free regions. # Problematic frame: # V [libjvm.so+0xc5a27c] void G1AdjustClosure::adjust_pointer<oop>(oop*)+0x5fc message shows that we are processing oopstorage when finding a bad oop: #27 0x00007fab25e5c2c6 in G1FullGCHeapRegionAttr::is_compacting (obj=<optimized out>, this=<optimized out>) at /src/hotspot/share/gc/g1/g1FullGCHeapRegionAttr.hpp:69 #28 G1FullCollector::is_compacting (obj=..., obj=..., this=0x7faaf532e490) at /src/hotspot/share/gc/g1/g1FullCollector.inline.hpp:35 #29 G1AdjustClosure::adjust_pointer<oop> (this=<optimized out>, p=0x7fab20368880) at /src/hotspot/share/gc/g1/g1FullGCOopClosures.inline.hpp:74 #30 0x00007fab25e699dd in G1AdjustClosure::do_oop_work<oop> (p=0x7fab20368880, this=0x7faaf532e310) at /src/hotspot/share/gc/g1/g1FullGCOopClosures.hpp:86 #31 G1AdjustClosure::do_oop (p=0x7fab20368880, this=0x7faaf532e310) at /src/hotspot/share/gc/g1/g1FullGCOopClosures.inline.hpp:95 #32 WeakProcessor::CountingClosure<AlwaysTrueClosure, G1AdjustClosure>::do_oop (p=0x7fab20368880, this=<synthetic pointer>) at /src/hotspot/share/gc/shared/weakProcessor.inline.hpp:65 #33 OopStorage::OopFn<WeakProcessor::CountingClosure<AlwaysTrueClosure, G1AdjustClosure> >::operator()<oop*> (ptr=0x7fab20368880, this=<synthetic pointer>) at /src/hotspot/share/gc/shared/oopStorage.inline.hpp:240 #34 OopStorage::BasicParState::AlwaysTrueFn<OopStorage::OopFn<WeakProcessor::CountingClosure<AlwaysTrueClosure, G1AdjustClosure> > >::operator()<oop*> (ptr=0x7fab20368880, this=<synthetic pointer>) at /src/hotspot/share/gc/shared/oopStorageParState.inline.hpp:42 #35 OopStorage::Block::iterate_impl<OopStorage::BasicParState::AlwaysTrueFn<OopStorage::OopFn<WeakProcessor::CountingClosure<AlwaysTrueClosure, G1AdjustClosure> > >, OopStorage::Block*> (block=0x7fab20368880, f=...) at /src/hotspot/share/gc/shared/oopStorage.inline.hpp:337 #36 OopStorage::Block::iterate<OopStorage::BasicParState::AlwaysTrueFn<OopStorage::OopFn<WeakProcessor::CountingClosure<AlwaysTrueClosure, G1AdjustClosure> > > > (f=..., this=0x7fab20368880) at /src/hotspot/share/gc/shared/oopStorage.inline.hpp:346 #37 OopStorage::BasicParState::iterate<false, OopStorage::OopFn<WeakProcessor::CountingClosure<AlwaysTrueClosure, G1AdjustClosure> > > (f=..., this=0x7faaf532e110) at /src/hotspot/share/gc/shared/oopStorageParState.inline.hpp:63 #38 OopStorage::ParState<false, false>::iterate<OopStorage::OopFn<WeakProcessor::CountingClosure<AlwaysTrueClosure, G1AdjustClosure> > > (f=..., this=0x7faaf532e110) at /src/hotspot/share/gc/shared/oopStorageParState.inline.hpp:82 #39 OopStorage::ParState<false, false>::oops_do<WeakProcessor::CountingClosure<AlwaysTrueClosure, G1AdjustClosure> > (cl=<synthetic pointer>, this=0x7faaf532e110) at /src/hotspot/share/gc/shared/oopStorageParState.inline.hpp:87 #40 WeakProcessor::Task::work<AlwaysTrueClosure, G1AdjustClosure> (this=this@entry=0x7faaf532e0b0, worker_id=worker_id@entry=0, is_alive=is_alive@entry=0x7faaf41f7ce8, keep_alive=keep_alive@entry=0x7faaf532e310) at /src/hotspot/share/gc/shared/weakProcessor.inline.hpp:91 #41 0x00007fab25e659a3 in G1FullGCAdjustTask::work (this=0x7faaf532df30, worker_id=0) at /src/hotspot/share/gc/g1/g1FullGCAdjustTask.cpp:106 ConcurrentMark::_has_aborted = true, so maybe that full gc aborted the conc mark as g1CollectorState::_mark_or_rebuild_in_progress = true too. That object reference is 0xffe56bd8, that points to | 13|0x0000000084b00000, 0x0000000084b00000, 0x0000000084c00000| 0%| F| |TAMS 0x0000000084b00000, 0x0000000084b00000| Untracked | 14|0x0000000084c00000, 0x0000000084c00000, 0x0000000084d00000| 0%| F| |TAMS 0x0000000084c00000, 0x0000000084c00000| Untracked |1984|0x00000000ffe00000, 0x00000000ffe00000, 0x00000000fff00000| 0%| F| |TAMS 0x00000000ffe00000, 0x00000000ffe00000| Untracked |1985|0x00000000fff00000, 0x00000000fff76000, 0x0000000100000000| 46%|CA| |TAMS 0x00000000fff76000, 0x00000000fff76000| Untracked I.e. a former Open Archive region (it's number is 1984, so highly likely an OA region) that is now a free region. Some issue with reclaiming completely empty OA regions?
20-09-2021
There are now multiple different crashes in this test linked to his bug: # assert(G1CollectedHeap::heap()->is_in_reserved(forwardee)) failed: should be in object space Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xadd241] os::platform_print_native_stack+0xf1 (os_windows_x86.cpp:235) V [jvm.dll+0xcfbe05] VMError::report+0x1005 (vmError.cpp:742) V [jvm.dll+0xcfd7ae] VMError::report_and_die+0x7fe (vmError.cpp:1552) V [jvm.dll+0xcfdf34] VMError::report_and_die+0x64 (vmError.cpp:1333) V [jvm.dll+0x4cda67] report_vm_error+0xb7 (debug.cpp:282) V [jvm.dll+0x5d0c8a] G1AdjustClosure::adjust_pointer<oopDesc *>+0x20a (g1FullGCOopClosures.inline.hpp:91) --- # assert(check_alignment(result)) failed: address not aligned: 0x000000082f617661 Stack: [0x000000a83b800000,0x000000a83b900000] Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xadd241] os::platform_print_native_stack+0xf1 (os_windows_x86.cpp:235) V [jvm.dll+0xcfbe05] VMError::report+0x1005 (vmError.cpp:742) V [jvm.dll+0xcfd7ae] VMError::report_and_die+0x7fe (vmError.cpp:1552) V [jvm.dll+0xcfdf34] VMError::report_and_die+0x64 (vmError.cpp:1333) V [jvm.dll+0x4cda67] report_vm_error+0xb7 (debug.cpp:282) V [jvm.dll+0x1dbc4] CompressedKlassPointers::decode_not_null+0x84 (compressedOops.inline.hpp:129) V [jvm.dll+0x5d78dc] HeapRegion::apply_to_marked_objects<G1FullGCPrepareTask::G1PrepareCompactLiveClosure>+0x12c (heapRegion.inline.hpp:245) V [jvm.dll+0x5d80b9] G1FullGCPrepareTask::G1CalculatePointersClosure::prepare_for_compaction+0xa9 (g1FullGCPrepareTask.cpp:200) V [jvm.dll+0x5d7dbe] G1FullGCPrepareTask::G1CalculatePointersClosure::do_heap_region+0x10e (g1FullGCPrepareTask.cpp:59) V [jvm.dll+0x6724b6] HeapRegionManager::par_iterate+0x156 (heapRegionManager.cpp:608) V [jvm.dll+0x5d85b1] G1FullGCPrepareTask::work+0xa1 (g1FullGCPrepareTask.cpp:122) V [jvm.dll+0xd41dea] GangWorker::loop+0x8a (workgroup.cpp:239) --- # assert(!is_free(obj)) failed: Should not have objects in free regions. # Problematic frame: # V [libjvm.so+0xc5a27c] void G1AdjustClosure::adjust_pointer<oop>(oop*)+0x5fc 9 failures over last 3 days
20-09-2021