We hit the debug memory limit in PhaseChaitin::Split by removing the temporary flag -XX:CompileCommand=MemLimit,*.*,0 from VectorReplicateLongSpecialImmTest.java and running the below command on aarch64.

  make test TEST=compiler/vectorapi/VectorReplicateLongSpecialImmTest.java JTREG='JAVA_OPTIONS=-XX:-UseTLAB'

The issue triggers after the fix for JDK-8324345, which results in skipping escape analysis in more cases than before. Potentially, skipping escape analysis and leaving a larger set of nodes could explain the present issue.

After fixing the (potential) issue, please remove the flag -XX:CompileCommand=MemLimit,*.*,0 from VectorReplicateLongSpecialImmTest.java.


#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/scratch/dlunden/jdk/open/src/hotspot/share/compiler/compilationMemoryStatistic.cpp:568), pid=677343, tid=677364
#  fatal error: c2 compiler/vectorapi/VectorReplicateLongSpecialImmTest::testReplicateL_Imm(()V): Hit MemLimit (limit: 1073741824 now: 1073748984)
#
# JRE version: Java(TM) SE Runtime Environment (24.0) (slowdebug build 24-internal-2024-08-05-1215129.dlunden...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (slowdebug 24-internal-2024-08-05-1215129.dlunden..., mixed mode, sharing, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Core dump will be written. Default location: /scratch/dlunden/jdk/build/linux-aarch64-slowdebug/test-support/jtreg_open_test_hotspot_jtreg_compiler_vectorapi_VectorReplicateLongSpecialImmTest_java/scratch/0/core.677343
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0000fffbfc1af270):  JavaThread "C2 CompilerThread3" daemon [_thread_in_native, id=677364, stack(0x0000fffc19710000,0x0000fffc19910000) (2048K)]


Current CompileTask:
C2:100677 1365 %           compiler.vectorapi.VectorReplicateLongSpecialImmTest::testReplicateL_Imm @ 2 (3851 bytes)

Stack: [0x0000fffc19710000,0x0000fffc19910000],  sp=0x0000fffc1990a540,  free space=2025k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x15b0e54]  VMError::report_and_die(int, char const*, char const*, std::__va_list, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x92c  (compilationMemoryStatistic.cpp:568)
V  [libjvm.so+0x9d6f78]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0x0  (debug.cpp:214)
V  [libjvm.so+0x9226f8]  CompilationMemoryStatistic::on_arena_change(long, Arena const*)+0x368  (compilationMemoryStatistic.cpp:568)
V  [libjvm.so+0x64866c]  Arena::set_size_in_bytes(unsigned long)+0xf8  (arena.cpp:266)
V  [libjvm.so+0x648828]  Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x124  (arena.cpp:308)
V  [libjvm.so+0x48d6a4]  Arena::internal_amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)+0x11c  (arena.hpp:134)
V  [libjvm.so+0x61b4c0]  Arena::Amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)+0xc4  (arena.hpp:155)
V  [libjvm.so+0x138bd84]  ResourceArea::allocate_bytes(unsigned long, AllocFailStrategy::AllocFailEnum)+0x2c  (resourceArea.inline.hpp:35)
V  [libjvm.so+0x1388088]  PhaseChaitin::Split(unsigned int, ResourceArea*)+0x374  (reg_split.cpp:555)
V  [libjvm.so+0x835afc]  PhaseChaitin::Register_Allocate()+0x3f0  (chaitin.cpp:481)
V  [libjvm.so+0x9372c4]  Compile::Code_Gen()+0x248  (compile.cpp:2966)
V  [libjvm.so+0x92d740]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x11f8  (compile.cpp:885)
V  [libjvm.so+0x7e5f78]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x1b4  (c2compiler.cpp:142)
V  [libjvm.so+0x951a98]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x8dc  (compileBroker.cpp:2303)
V  [libjvm.so+0x950688]  CompileBroker::compiler_thread_loop()+0x3a4  (compileBroker.cpp:1961)
V  [libjvm.so+0x972360]  CompilerThread::thread_entry(JavaThread*, JavaThread*)+0xb0  (compilerThread.cpp:65)
V  [libjvm.so+0xdb0200]  JavaThread::thread_main_inner()+0x188  (javaThread.cpp:757)
V  [libjvm.so+0xdb006c]  JavaThread::run()+0x208  (javaThread.cpp:742)
V  [libjvm.so+0x14fa214]  Thread::call_run()+0x1d8  (thread.cpp:225)
V  [libjvm.so+0x12bf274]  thread_native_entry(Thread*)+0x1d0  (os_linux.cpp:858)
C  [libpthread.so.0+0x7950]  start_thread+0x190
Lock stack of current Java thread (top to bottom):