The attached Java Fuzzer test hit the memory limit assert. I then run it with the new CompileTaskTimeout and it timed out after 10s. I reduced the test a bit (Reduced.java) which now still takes 10s but will succeed a few seconds later.

There still seems to be an inefficiency which we should investigate further.

# Intermittent - depends on StressLoopPeeling taking the "right" decisions to peel. Otherwise, the compilation is completed instantly. By running with TraceLoopOpts, one can see where it suddenly becomes stuck.
$ java -XX:CompileCommand=compileonly,*Reduced*::test -XX:-TieredCompilation -Xcomp -XX:CompileCommand=dontinline,*::* -XX:+StressLoopPeeling -XX:PerMethodTrapLimit=0 -XX:+VerifyLoopOptimizations -XX:CompileTaskTimeout=10000 -XX:+TraceLoopOpts Reduced.java

$ java -XX:CompileOnly=Test::* -XX:-TieredCompilation -Xcomp -XX:CompileCommand=dontinline,*::* -XX:+StressLoopPeeling -XX:PerMethodTrapLimit=0 -XX:+VerifyLoopOptimizations -XX:CompileCommand=printcompilation,*::* -XX:CompileTaskTimeout=10000 -XX:+TraceLoopOpts Test.java


#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/home/chagedor/jdk/open/src/hotspot/os/linux/compilerThreadTimeout_linux.cpp:47), pid=933639, tid=933654
#  assert(false) failed: compile task 98 (Reduced.test()V) timed out after 10000 ms
#
# JRE version: Java(TM) SE Runtime Environment (26.0) (slowdebug build 26-internal-2025-08-29-0744563.chagedor...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (slowdebug 26-internal-2025-08-29-0744563.chagedor..., compiled mode, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xaae5f1]  CompilerThreadTimeoutLinux::compiler_signal_handler(int, siginfo*, void*)+0xe3


Stuck at timeout:
V  [libjvm.so+0xaae5f1]  CompilerThreadTimeoutLinux::compiler_signal_handler(int, siginfo*, void*)+0xe3  (compilerThreadTimeout_linux.cpp:47)
V  [libjvm.so+0xaae50b]  compiler_signal_handler(int, siginfo*, void*)+0x35  (compilerThreadTimeout_linux.cpp:37)
C  [libc.so.6+0x3ebf0]
V  [libjvm.so+0x13b82e9]  PhaseIdealLoop::get_late_ctrl_with_anti_dep(LoadNode*, Node*, Node*)+0xb8d  (loopnode.cpp:6510)
V  [libjvm.so+0x13b76e2]  PhaseIdealLoop::get_late_ctrl(Node*, Node*)+0x132  (loopnode.cpp:6435)
V  [libjvm.so+0x13b9404]  PhaseIdealLoop::build_loop_late_post_work(Node*, bool)+0x42a  (loopnode.cpp:6823)
V  [libjvm.so+0x13b8fd6]  PhaseIdealLoop::build_loop_late_post(Node*)+0x28  (loopnode.cpp:6711)
V  [libjvm.so+0x13b8bb8]  PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x272  (loopnode.cpp:6656)
V  [libjvm.so+0x13b1f12]  PhaseIdealLoop::build_and_optimize()+0xbfc  (loopnode.cpp:5093)
V  [libjvm.so+0xa7df1e]  PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, PhaseIdealLoop const*)+0x14e  (loopnode.hpp:1204)
V  [libjvm.so+0x13b3195]  PhaseIdealLoop::verify() const+0x135  (loopnode.cpp:5332)
V  [libjvm.so+0x13c623e]  PhaseIdealLoop::split_if_with_blocks_post(Node*)+0x8c4  (loopopts.cpp:1527)
V  [libjvm.so+0x13c8c4b]  PhaseIdealLoop::split_if_with_blocks(VectorSet&, Node_Stack&)+0x1f3  (loopopts.cpp:1984)
V  [libjvm.so+0x13b2894]  PhaseIdealLoop::build_and_optimize()+0x157e  (loopnode.cpp:5206)
V  [libjvm.so+0xa7ddc5]  PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, LoopOptsMode)+0x191  (loopnode.hpp:1189)
V  [libjvm.so+0xa7e264]  PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x10c  (loopnode.hpp:1283)
V  [libjvm.so+0xa6c017]  Compile::optimize_loops(PhaseIterGVN&, LoopOptsMode)+0x9d  (compile.cpp:2254)
V  [libjvm.so+0xa6d711]  Compile::Optimize()+0x1519  (compile.cpp:2503)
V  [libjvm.so+0xa648a2]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1916  (compile.cpp:859)
V  [libjvm.so+0x90c489]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x48f  (c2compiler.cpp:147)
V  [libjvm.so+0xa8cddd]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0xe01  (compileBroker.cpp:2340)
V  [libjvm.so+0xa8b1d6]  CompileBroker::compiler_thread_loop()+0x596  (compileBroker.cpp:1984)
V  [libjvm.so+0xaae1c1]  CompilerThread::thread_entry(JavaThread*, JavaThread*)+0x89  (compilerThread.cpp:69)
V  [libjvm.so+0xf94cfc]  JavaThread::thread_main_inner()+0x1b2  (javaThread.cpp:775)
V  [libjvm.so+0xf94b47]  JavaThread::run()+0x1e3  (javaThread.cpp:760)
V  [libjvm.so+0x189e66d]  Thread::call_run()+0x1b9  (thread.cpp:243)
V  [libjvm.so+0x159bbd1]  thread_native_entry(Thread*)+0x1f5  (os_linux.cpp:868)


Usually, the failure mode looks like this (JDK-8367039):

# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/workspace/open/src/hotspot/share/compiler/compilationMemoryStatistic.cpp:935), pid=2646382, tid=2646411
# fatal error: c2 (8) Test::mainTest(([Ljava/lang/String;)V): Hit MemLimit - limit: 1073741824 now: 1073762896
#

Current CompileTask:
C2:104916 8 !b Test::mainTest (1934 bytes)

Stack: [0x00007f423d6a5000,0x00007f423d7a5000], sp=0x00007f423d79ebd0, free space=998k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x1c57423] VMError::report(outputStream*, bool)+0x3023 (compilationMemoryStatistic.cpp:935)
V [libjvm.so+0x1c5ae13] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void const*, void const*, char const*, int, unsigned long)+0x5a3 (vmError.cpp:1875)
V [libjvm.so+0xc57a4b] report_fatal(VMErrorType, char const*, int, char const*, ...)+0x10b (debug.cpp:222)
V [libjvm.so+0xb55fff] CompilationMemoryStatistic::on_arena_chunk_allocation(unsigned long, int, unsigned long*)+0x6ef (compilationMemoryStatistic.cpp:935)
V [libjvm.so+0x713556] ChunkPool::allocate_chunk(Arena*, unsigned long, AllocFailStrategy::AllocFailEnum)+0x166 (arena.cpp:195)
V [libjvm.so+0x7143b4] Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x74 (arena.cpp:329)
V [libjvm.so+0x7146e9] Arena::Arealloc(void*, unsigned long, unsigned long, AllocFailStrategy::AllocFailEnum)+0x1d9 (arena.hpp:153)
V [libjvm.so+0x1758b53] Node_Array::grow(unsigned int)+0x53 (node.cpp:2805)
V [libjvm.so+0x9cc04b] Unique_Node_List::push(Node*)+0xcb (node.hpp:1639)
V [libjvm.so+0x158ce75] PhaseIdealLoop::get_late_ctrl_with_anti_dep(LoadNode*, Node*, Node*)+0x255 (loopnode.cpp:6456)
V [libjvm.so+0x158e351] PhaseIdealLoop::get_late_ctrl(Node*, Node*)+0xf1 (loopnode.cpp:6435)
V [libjvm.so+0x158e582] PhaseIdealLoop::build_loop_late_post_work(Node*, bool)+0x1f2 (loopnode.cpp:6823)
V [libjvm.so+0x158ee83] PhaseIdealLoop::build_loop_late(VectorSet&, Node_List&, Node_Stack&)+0x1a3 (loopnode.cpp:6711)
V [libjvm.so+0x158f8d7] PhaseIdealLoop::build_and_optimize()+0x7e7 (loopnode.cpp:5093)
V [libjvm.so+0x15909b1] PhaseIdealLoop::verify() const+0x341 (loopnode.hpp:1204)
V [libjvm.so+0x159f2ed] PhaseIdealLoop::split_if_with_blocks(VectorSet&, Node_Stack&)+0xbd (loopopts.cpp:1984)
V [libjvm.so+0x1590456] PhaseIdealLoop::build_and_optimize()+0x1366 (loopnode.cpp:5206)
V [libjvm.so+0xb8bba3] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x4f3 (loopnode.hpp:1189)
V [libjvm.so+0xb82d86] Compile::optimize_loops(PhaseIterGVN&, LoopOptsMode)+0xb6 (compile.cpp:2254)
V [libjvm.so+0xb85b41] Compile::Optimize()+0xcc1 (compile.cpp:2503)
V [libjvm.so+0xb88c5f] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1f8f (compile.cpp:859)
V [libjvm.so+0x9adcb6] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x466 (c2compiler.cpp:147)
V [libjvm.so+0xb98118] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xb48 (compileBroker.cpp:2342)
V [libjvm.so+0xb992a0] CompileBroker::compiler_thread_loop()+0x530 (compileBroker.cpp:1986)
V [libjvm.so+0x10f944b] JavaThread::thread_main_inner()+0x13b (javaThread.cpp:775)
V [libjvm.so+0x1b618c6] Thread::call_run()+0xb6 (thread.cpp:243)
V [libjvm.so+0x17d4e18] thread_native_entry(Thread*)+0x128 (os_linux.cpp:892