Bug ID: JDK-8311843 GenShen: assertion failed "Old generation affiliated regions must be less than capacity"

Type: Bug
Component: hotspot
Sub-Component: gc
Affected Version: repo-shenandoah

Priority: P2
Status: Open
Resolution: Unresolved

Submitted: 2023-07-10
Updated: 2023-08-28

This assertion failure has been observed in `TestAllocOutOfMemory.java` and `TestThreadFailure.java`.

```
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/Users/runner/work/shenandoah/shenandoah/src/hotspot/share/gc/shenandoah/shenandoahFullGC.cpp:184), pid=6021, tid=18435
#  assert(heap->old_generation()->used_regions_size() <= heap->old_generation()->max_capacity()) failed: Old generation affiliated regions must be less than capacity
#
# JRE version: OpenJDK Runtime Environment (22.0) (fastdebug build 22-internal-earthling-amzn-5c2860661e52317ff513ca58924d71f1d8fff596)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 22-internal-earthling-amzn-5c2860661e52317ff513ca58924d71f1d8fff596, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, shenandoah gc, bsd-amd64)
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
```

A pull request was submitted for review. URL: https://git.openjdk.org/shenandoah/pull/311 Date: 2023-08-25 17:45:44 +0000
25-08-2023
Reassigned to William as he's working on a fix.
22-08-2023
This seems to be the crux of the issue. It appears as if, at least in my set up, I need `-ClassUnloading +ClassUnloadingWithConcMark +ShenandoahGenerationalAdaptiveTenuring` to be true and then this happens reliably. Need to look at the combination of class unloading flags to make sure that we don't end up with wires crossed. The test explicitly wants to check for this, per test modes and comments for each of the test modes, requiring the global setting `-ClassUnloading` to override local settings that might conflict. (Note to self: I'd like to also verify that configuration doesn't change if order of these flags is reversed etc.). ``` # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (/home/ysr/workplace/ysr/genshen-ysr/shenandoah/src/hotspot/share/gc/shenandoah/shenandoahVerifier.cpp:433), pid=40059, tid=40069 # guarantee(stats.span() <= generation_capacity) failed: After Full GC: generation (OLD) size spanned by regions (479) must not exceed current capacity (119M) # # JRE version: OpenJDK Runtime Environment (22.0) (slowdebug build 22-internal-adhoc.ysr.shenandoah) # Java VM: OpenJDK 64-Bit Server VM (slowdebug 22-internal-adhoc.ysr.shenandoah, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, shenandoah gc, linux-amd64) # Problematic frame: # V [libjvm.so+0x1436c97] ShenandoahGenerationStatsClosure::validate_usage(bool, char const, ShenandoahGeneration, ShenandoahCalculateRegionStatsClosure&)+0x249 # # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # --------------- S U M M A R Y ------------ Command Line: -Xmx128m -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+UseShenandoahGC -Xlog:gc=info -XX:+ShenandoahVerify -XX:+VerifyBeforeGC -XX:+VerifyAfterGC -XX:ShenandoahGCMode=generational -XX:ShenandoahGCHeuristics=adaptive -XX:-ClassUnloading -XX:+ClassUnloadingWithConcurrentMark TestClassLoaderLeak test Host: dev-dsk-ysr-2a-87502c94.us-west-2.amazon.com, Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz, 40 cores, 157G, Amazon Linux release 2 (Karoo) Time: Mon Aug 21 18:59:28 2023 UTC elapsed time: 2.399857 seconds (0d 0h 0m 2s) --------------- T H R E A D --------------- Current thread (0x00007f2a9c209430): VMThread "VM Thread" [id=40069, stack(0x00007f2a821fd000,0x00007f2a822fd000) (1024K)] Stack: [0x00007f2a821fd000,0x00007f2a822fd000], sp=0x00007f2a822fafe0, free space=1015k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x1436c97] ShenandoahGenerationStatsClosure::validate_usage(bool, char const, ShenandoahGeneration, ShenandoahCalculateRegionStatsClosure&)+0x249 (shenandoahVerifier.cpp:433) V [libjvm.so+0x1433a3e] ShenandoahVerifier::verify_at_safepoint(char const, ShenandoahVerifier::VerifyRememberedSet, ShenandoahVerifier::VerifyForwarded, ShenandoahVerifier::VerifyMarked, ShenandoahVerifier::VerifyCollectionSet, ShenandoahVerifier::VerifyLiveness, ShenandoahVerifier::VerifyRegions, ShenandoahVerifier::VerifySize, ShenandoahVerifier::VerifyGCState)+0x972 (shenandoahVerifier.cpp:918) V [libjvm.so+0x143449b] ShenandoahVerifier::verify_after_fullgc()+0x3d (shenandoahVerifier.cpp:1173) V [libjvm.so+0x1374266] ShenandoahFullGC::do_it(GCCause::Cause)+0x9b4 (shenandoahFullGC.cpp:383) V [libjvm.so+0x13735ae] ShenandoahFullGC::op_full(GCCause::Cause)+0x2d4 (shenandoahFullGC.cpp:190) V [libjvm.so+0x1367757] ShenandoahDegenGC::op_degenerated_futile()+0x35 (shenandoahDegeneratedGC.cpp:479) V [libjvm.so+0x13671e4] ShenandoahDegenGC::op_degenerated()+0xb48 (shenandoahDegeneratedGC.cpp:349) V [libjvm.so+0x136663c] ShenandoahDegenGC::entry_degenerated()+0xec (shenandoahDegeneratedGC.cpp:86) V [libjvm.so+0x1432a23] VM_ShenandoahDegeneratedGC::doit()+0x3b (shenandoahVMOperations.cpp:67) V [libjvm.so+0x162ea3e] VM_Operation::evaluate()+0xda (vmOperations.cpp:71) V [libjvm.so+0x16985a6] VMThread::evaluate_operation(VM_Operation)+0x62 (vmThread.cpp:281) V [libjvm.so+0x1698e86] VMThread::inner_execute(VM_Operation)+0x3c4 (vmThread.cpp:435) V [libjvm.so+0x169927b] VMThread::loop()+0x131 (vmThread.cpp:502) V [libjvm.so+0x169816c] VMThread::run()+0xfa (vmThread.cpp:175) V [libjvm.so+0x157e442] Thread::call_run()+0x1a4 (thread.cpp:217) V [libjvm.so+0x1204afa] thread_native_entry(Thread*)+0x1b9 (os_linux.cpp:783) ```
21-08-2023
Reprioritized to P2, since it's a fastdebug jtreg failure and indicates an invariant violation that might compromise subsequent operation.
18-08-2023
This appears to be an issue with the region affiliation accounting by a full gc. A jtr file is attached from TestClassLoaderLeak with genshen w/adaptive tenuring enabled: ``` CONF=fastdebug make test TEST="test/hotspot/jtreg/gc/shenandoah/oom/TestClassLoaderLeak.java" ```
18-08-2023
I seem to now have a reproducible test case. I'll pick this up.
15-08-2023
Recently observed in GHA testing at https://github.com/ysramakrishna/shenandoah/actions/runs/5694503099 with the following tests: * gc/shenandoah/oom/TestClassLoaderLeak * gc/shenandoah/oom/TestThreadFailure Seems to be pretty difficult to reproduce on my local repo so far.
28-07-2023