The attached Java Fuzzer test hit the memory limit assert. I then run it with the new CompileTaskTimeout and it timed out after 10s. I reduced the test a bit (Reduced.java) which now still takes 10s but will succeed a few seconds later.
There still seems to be an inefficiency which we should investigate further.
# Intermittent - depends on StressLoopPeeling taking the "right" decisions to peel. Otherwise, the compilation is completed instantly. By running with TraceLoopOpts, one can see where it suddenly becomes stuck.
$ java -XX:CompileCommand=compileonly,*Reduced*::test -XX:-TieredCompilation -Xcomp -XX:CompileCommand=dontinline,*::* -XX:+StressLoopPeeling -XX:PerMethodTrapLimit=0 -XX:+VerifyLoopOptimizations -XX:CompileTaskTimeout=10000 -XX:+TraceLoopOpts Reduced.java
$ java -XX:CompileOnly=Test::* -XX:-TieredCompilation -Xcomp -XX:CompileCommand=dontinline,*::* -XX:+StressLoopPeeling -XX:PerMethodTrapLimit=0 -XX:+VerifyLoopOptimizations -XX:CompileCommand=printcompilation,*::* -XX:CompileTaskTimeout=10000 -XX:+TraceLoopOpts Test.java
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/home/chagedor/jdk/open/src/hotspot/os/linux/compilerThreadTimeout_linux.cpp:47), pid=933639, tid=933654
# assert(false) failed: compile task 98 (Reduced.test()V) timed out after 10000 ms
#
# JRE version: Java(TM) SE Runtime Environment (26.0) (slowdebug build 26-internal-2025-08-29-0744563.chagedor...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (slowdebug 26-internal-2025-08-29-0744563.chagedor..., compiled mode, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0xaae5f1] CompilerThreadTimeoutLinux::compiler_signal_handler(int, siginfo*, void*)+0xe3
Stuck at timeout:
V [libjvm.so+0xaae5f1] CompilerThreadTimeoutLinux::compiler_signal_handler(int, siginfo*, void*)+0xe3 (compilerThreadTimeout_linux.cpp:47)
V [libjvm.so+0xaae50b] compiler_signal_handler(int, siginfo*, void*)+0x35 (compilerThreadTimeout_linux.cpp:37)
C [libc.so.6+0x3ebf0]
V [libjvm.so+0x13b82e9] PhaseIdealLoop::get_late_ctrl_with_anti_dep(LoadNode*, Node*, Node*)+0xb8d (loopnode.cpp:6510)
V [libjvm.so+0x13b76e2] PhaseIdealLoop::get_late_ctrl(Node*, Node*)+0x132 (loopnode.cpp:6435)
V [libjvm.so+0x13b9404] PhaseIdealLoop::build_loop_late_post_work(Node*, bool)+0x42a (loopnode.cpp:6823)
V [libjvm.so+0x13b8fd6] PhaseIdealLoop::build_loop_late_post(Node*)+0x28 (loopnode.cpp:6711)
V [libjvm.so+0x13b8bb8] PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x272 (loopnode.cpp:6656)
V [libjvm.so+0x13b1f12] PhaseIdealLoop::build_and_optimize()+0xbfc (loopnode.cpp:5093)
V [libjvm.so+0xa7df1e] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, PhaseIdealLoop const*)+0x14e (loopnode.hpp:1204)
V [libjvm.so+0x13b3195] PhaseIdealLoop::verify() const+0x135 (loopnode.cpp:5332)
V [libjvm.so+0x13c623e] PhaseIdealLoop::split_if_with_blocks_post(Node*)+0x8c4 (loopopts.cpp:1527)
V [libjvm.so+0x13c8c4b] PhaseIdealLoop::split_if_with_blocks(VectorSet&, Node_Stack&)+0x1f3 (loopopts.cpp:1984)
V [libjvm.so+0x13b2894] PhaseIdealLoop::build_and_optimize()+0x157e (loopnode.cpp:5206)
V [libjvm.so+0xa7ddc5] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, LoopOptsMode)+0x191 (loopnode.hpp:1189)
V [libjvm.so+0xa7e264] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x10c (loopnode.hpp:1283)
V [libjvm.so+0xa6c017] Compile::optimize_loops(PhaseIterGVN&, LoopOptsMode)+0x9d (compile.cpp:2254)
V [libjvm.so+0xa6d711] Compile::Optimize()+0x1519 (compile.cpp:2503)
V [libjvm.so+0xa648a2] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1916 (compile.cpp:859)
V [libjvm.so+0x90c489] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x48f (c2compiler.cpp:147)
V [libjvm.so+0xa8cddd] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xe01 (compileBroker.cpp:2340)
V [libjvm.so+0xa8b1d6] CompileBroker::compiler_thread_loop()+0x596 (compileBroker.cpp:1984)
V [libjvm.so+0xaae1c1] CompilerThread::thread_entry(JavaThread*, JavaThread*)+0x89 (compilerThread.cpp:69)
V [libjvm.so+0xf94cfc] JavaThread::thread_main_inner()+0x1b2 (javaThread.cpp:775)
V [libjvm.so+0xf94b47] JavaThread::run()+0x1e3 (javaThread.cpp:760)
V [libjvm.so+0x189e66d] Thread::call_run()+0x1b9 (thread.cpp:243)
V [libjvm.so+0x159bbd1] thread_native_entry(Thread*)+0x1f5 (os_linux.cpp:868)
Usually, the failure mode looks like this (JDK-8367039):
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/workspace/open/src/hotspot/share/compiler/compilationMemoryStatistic.cpp:935), pid=2646382, tid=2646411
# fatal error: c2 (8) Test::mainTest(([Ljava/lang/String;)V): Hit MemLimit - limit: 1073741824 now: 1073762896
#
Current CompileTask:
C2:104916 8 !b Test::mainTest (1934 bytes)
Stack: [0x00007f423d6a5000,0x00007f423d7a5000], sp=0x00007f423d79ebd0, free space=998k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x1c57423] VMError::report(outputStream*, bool)+0x3023 (compilationMemoryStatistic.cpp:935)
V [libjvm.so+0x1c5ae13] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void const*, void const*, char const*, int, unsigned long)+0x5a3 (vmError.cpp:1875)
V [libjvm.so+0xc57a4b] report_fatal(VMErrorType, char const*, int, char const*, ...)+0x10b (debug.cpp:222)
V [libjvm.so+0xb55fff] CompilationMemoryStatistic::on_arena_chunk_allocation(unsigned long, int, unsigned long*)+0x6ef (compilationMemoryStatistic.cpp:935)
V [libjvm.so+0x713556] ChunkPool::allocate_chunk(Arena*, unsigned long, AllocFailStrategy::AllocFailEnum)+0x166 (arena.cpp:195)
V [libjvm.so+0x7143b4] Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x74 (arena.cpp:329)
V [libjvm.so+0x7146e9] Arena::Arealloc(void*, unsigned long, unsigned long, AllocFailStrategy::AllocFailEnum)+0x1d9 (arena.hpp:153)
V [libjvm.so+0x1758b53] Node_Array::grow(unsigned int)+0x53 (node.cpp:2805)
V [libjvm.so+0x9cc04b] Unique_Node_List::push(Node*)+0xcb (node.hpp:1639)
V [libjvm.so+0x158ce75] PhaseIdealLoop::get_late_ctrl_with_anti_dep(LoadNode*, Node*, Node*)+0x255 (loopnode.cpp:6456)
V [libjvm.so+0x158e351] PhaseIdealLoop::get_late_ctrl(Node*, Node*)+0xf1 (loopnode.cpp:6435)
V [libjvm.so+0x158e582] PhaseIdealLoop::build_loop_late_post_work(Node*, bool)+0x1f2 (loopnode.cpp:6823)
V [libjvm.so+0x158ee83] PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x1a3 (loopnode.cpp:6711)
V [libjvm.so+0x158f8d7] PhaseIdealLoop::build_and_optimize()+0x7e7 (loopnode.cpp:5093)
V [libjvm.so+0x15909b1] PhaseIdealLoop::verify() const+0x341 (loopnode.hpp:1204)
V [libjvm.so+0x159f2ed] PhaseIdealLoop::split_if_with_blocks(VectorSet&, Node_Stack&)+0xbd (loopopts.cpp:1984)
V [libjvm.so+0x1590456] PhaseIdealLoop::build_and_optimize()+0x1366 (loopnode.cpp:5206)
V [libjvm.so+0xb8bba3] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x4f3 (loopnode.hpp:1189)
V [libjvm.so+0xb82d86] Compile::optimize_loops(PhaseIterGVN&, LoopOptsMode)+0xb6 (compile.cpp:2254)
V [libjvm.so+0xb85b41] Compile::Optimize()+0xcc1 (compile.cpp:2503)
V [libjvm.so+0xb88c5f] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1f8f (compile.cpp:859)
V [libjvm.so+0x9adcb6] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x466 (c2compiler.cpp:147)
V [libjvm.so+0xb98118] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xb48 (compileBroker.cpp:2342)
V [libjvm.so+0xb992a0] CompileBroker::compiler_thread_loop()+0x530 (compileBroker.cpp:1986)
V [libjvm.so+0x10f944b] JavaThread::thread_main_inner()+0x13b (javaThread.cpp:775)
V [libjvm.so+0x1b618c6] Thread::call_run()+0xb6 (thread.cpp:243)
V [libjvm.so+0x17d4e18] thread_native_entry(Thread*)+0x128 (os_linux.cpp:892