Please mention me in your PR, I spent a few hours extracting a nice JASM file from a nasty class-file for you, and produced this stand-alone reproducer.
The JASM file can probably be a bit further reduced, and maybe even a JAVA file extracted.
JDK11 does not seem to reproduce, but JDK17 does.
Could be related to JDK-8335709, and other infinite-loop bugs.
emanuel@emanuel-oracle:/oracle-work/triage$ java -jar ~/Documents/asmtools-7.0-build/release/lib/asmtools.jar jasm Y.jasm
emanuel@emanuel-oracle:/oracle-work/triage$ /oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/java -XX:CompileCommand=printcompilation,Y::* -XX:CompileCommand=compileonly,Y::test -Xcomp Y
CompileCommand: PrintCompilation Y.* bool PrintCompilation = true
CompileCommand: compileonly Y.test bool compileonly = true
493 26 b 3 Y::test (40 bytes)
494 27 b 4 Y::test (40 bytes)
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (/oracle-work/jdk-fork4/open/src/hotspot/share/opto/loopopts.cpp:1699), pid=3961958, tid=3961972
# assert(!n->is_Store() && !n->is_LoadStore()) failed: no node with a side effect
#
# JRE version: Java(TM) SE Runtime Environment (23.0) (fastdebug build 23-internal-2024-05-22-1054280.emanuel...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 23-internal-2024-05-22-1054280.emanuel..., compiled mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x12a956e] PhaseIdealLoop::try_sink_out_of_loop(Node*)+0xc8e
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /oracle-work/triage/core.3961958)
#
# An error report file with more information is saved as:
# /oracle-work/triage/hs_err_pid3961958.log
#
# Compiler replay data is saved as:
# /oracle-work/triage/replay_pid3961958.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)
Current CompileTask:
C2:519 27 b 4 Y::test (40 bytes)
Stack: [0x00007f8e89234000,0x00007f8e89335000], sp=0x00007f8e8932fd30, free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x12a956e] PhaseIdealLoop::try_sink_out_of_loop(Node*)+0xc8e (loopopts.cpp:1699)
V [libjvm.so+0x12a9643] PhaseIdealLoop::split_if_with_blocks_post(Node*)+0x83 (loopopts.cpp:1513)
V [libjvm.so+0x12a9f06] PhaseIdealLoop::split_if_with_blocks(VectorSet&, Node_Stack&)+0x96 (loopopts.cpp:1945)
V [libjvm.so+0x129c2e9] PhaseIdealLoop::build_and_optimize()+0xee9 (loopnode.cpp:4816)
V [libjvm.so+0x9eb1f0] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x390 (loopnode.hpp:1115)
V [libjvm.so+0x9e3f35] Compile::optimize_loops(PhaseIterGVN&, LoopOptsMode)+0x75 (compile.cpp:2188)
V [libjvm.so+0x9e6641] Compile::Optimize()+0xe21 (compile.cpp:2435)
V [libjvm.so+0x9e9c46] Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1af6 (compile.cpp:853)
V [libjvm.so+0x83e1f5] C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x1d5 (c2compiler.cpp:142)
V [libjvm.so+0x9f5a78] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x928 (compileBroker.cpp:2303)
V [libjvm.so+0x9f6708] CompileBroker::compiler_thread_loop()+0x478 (compileBroker.cpp:1961)
V [libjvm.so+0xe9f6bc] JavaThread::thread_main_inner()+0xcc (javaThread.cpp:759)
V [libjvm.so+0x17b2de6] Thread::call_run()+0xb6 (thread.cpp:225)
V [libjvm.so+0x149f1b7] thread_native_entry(Thread*)+0x127 (os_linux.cpp:846)
Registers:
RAX=0x00007f8eb9385000, RBX=0x00007f8e5c02cfd8, RCX=0x00007f8eb8a93499, RDX=0x00007f8eb8bb1f18
RSP=0x00007f8e8932fd30, RBP=0x00007f8e8932fe10, RSI=0x00000000000006a3, RDI=0x00007f8eb8bb14a8
R8 =0x00007f8e5806ec88, R9 =0x00007f8e5805b900, R10=0x0000000000000003, R11=0x00007f8eb9080f64
R12=0x00007f8e5806d9e0, R13=0x00007f8e89330140, R14=0x00007f8e5806d9e0, R15=0x00000000cccccccd
RIP=0x00007f8eb829456e, EFLAGS=0x0000000000010246, CSGSFS=0x002b000000000033, ERR=0x0000000000000006
TRAPNO=0x000000000000000e
Having a quick look with RR:
rr /oracle-work/jdk-fork4/build/linux-x64-slowdebug/jdk/bin/java -XX:CompileCommand=printcompilation,Y::* -XX:CompileCommand=compileonly,Y::test -Xcomp Y
(rr) p n->dump_bfs(4,0,"#c")
dist dump
---------------------------------------------
4 235 Loop === 235 86 380 [[ 235 89 284 412 ]] !orig=[87] !jvms: Y::test @ bci:0
3 412 NeverBranch === 235 [[ 413 414 ]]
2 413 CProj === 412 [[ 288 ]] #0
1 288 SafePoint === 413 1 89 1 1 284 1 92 1 92 133 [[ 281 356 351 389 ]] SafePoint !orig=134 !jvms: Y::test @ bci:5
0 389 StoreI === 288 281 149 92 [[ 280 279 ]] @java/lang/Class (java/io/Serializable,java/lang/constant/Constable,java/lang/reflect/AnnotatedElement,java/lang/invoke/TypeDescriptor,java/lang/reflect/GenericDeclaration,java/lang/reflect/Type,java/lang/invoke/TypeDescriptor$OfField):exact+112 *, name=x, idx=4; Memory: @java/lang/Class (java/io/Serializable,java/lang/constant/Constable,java/lang/reflect/AnnotatedElement,java/lang/invoke/TypeDescriptor,java/lang/reflect/GenericDeclaration,java/lang/reflect/Type,java/lang/invoke/TypeDescriptor$OfField):exact+112 *, name=x, idx=4; !orig=[150],[162],[197] !jvms: Y::test @ bci:9
This looks strongly like an infinite-loop bug, see the NeverBranch node.