Please mention me in the PR, I put a lot of work into getting a nice JASM reproducer from some nasty compiled class files.

emanuel@emanuel-oracle:/oracle-work/triage/next$ java -jar ~/Documents/asmtools-7.0-build/release/lib/asmtools.jar jasm X.jasm
emanuel@emanuel-oracle:/oracle-work/triage/next$ /oracle-work/jdk-fork4/build/linux-x64-debug/jdk/bin/java -XX:+UnlockExperimentalVMOptions -Xcomp -XX:PerMethodTrapLimit=0 -XX:PerMethodSpecTrapLimit=0 -XX:StressLongCountedLoop=2000000 -XX:CompileCommand=compileonly,X::* -XX:CompileCommand=printcompilation,X::* X
CompileCommand: compileonly X.* bool compileonly = true
CompileCommand: PrintCompilation X.* bool PrintCompilation = true
492   26    b  3       X::main (11 bytes)
493   27    b  4       X::main (11 bytes)
493   28    b  3       X::test (33 bytes)
494   29    b  4       X::test (33 bytes)
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/oracle-work/jdk-fork4/open/src/hotspot/share/opto/compile.cpp:3776), pid=3982454, tid=3982468
#  assert(!n->as_Loop()->is_loop_nest_inner_loop() || _loop_opts_cnt == 0) failed: should have been turned into a counted loop
#
# JRE version: Java(TM) SE Runtime Environment (23.0) (fastdebug build 23-internal-2024-05-22-1054280.emanuel...)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 23-internal-2024-05-22-1054280.emanuel..., compiled mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x9d8634]  Compile::final_graph_reshaping_main_switch(Node*, Final_Reshape_Counts&, unsigned int, Unique_Node_List&)+0x2144
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /oracle-work/triage/next/core.3982454)
#
# An error report file with more information is saved as:
# /oracle-work/triage/next/hs_err_pid3982454.log
#
# Compiler replay data is saved as:
# /oracle-work/triage/next/replay_pid3982454.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)

Current CompileTask:
C2:519   29    b  4       X::test (33 bytes)

Stack: [0x00007ff53c140000,0x00007ff53c241000],  sp=0x00007ff53c23be80,  free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x9d8634]  Compile::final_graph_reshaping_main_switch(Node*, Final_Reshape_Counts&, unsigned int, Unique_Node_List&)+0x2144  (compile.cpp:3776)
V  [libjvm.so+0x9d88ef]  Compile::final_graph_reshaping_impl(Node*, Final_Reshape_Counts&, Unique_Node_List&) [clone .part.0]+0x15f  (compile.cpp:3171)
V  [libjvm.so+0x9d91f4]  Compile::final_graph_reshaping_walk(Node_Stack&, Node*, Final_Reshape_Counts&, Unique_Node_List&)+0x174  (compile.cpp:3120)
V  [libjvm.so+0x9e4afd]  Compile::final_graph_reshaping()+0x53d  (compile.cpp:4059)
V  [libjvm.so+0x9e68dd]  Compile::Optimize()+0x10bd  (compile.cpp:2502)
V  [libjvm.so+0x9e9c46]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1af6  (compile.cpp:853)
V  [libjvm.so+0x83e1f5]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x1d5  (c2compiler.cpp:142)
V  [libjvm.so+0x9f5a78]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x928  (compileBroker.cpp:2303)
V  [libjvm.so+0x9f6708]  CompileBroker::compiler_thread_loop()+0x478  (compileBroker.cpp:1961)
V  [libjvm.so+0xe9f6bc]  JavaThread::thread_main_inner()+0xcc  (javaThread.cpp:759)
V  [libjvm.so+0x17b2de6]  Thread::call_run()+0xb6  (thread.cpp:225)
V  [libjvm.so+0x149f1b7]  thread_native_entry(Thread*)+0x127  (os_linux.cpp:846)


Having a quick look with RR:

rr /oracle-work/jdk-fork4/build/linux-x64-slowdebug/jdk/bin/java -XX:+UnlockExperimentalVMOptions -Xcomp -XX:PerMethodTrapLimit=0 -XX:PerMethodSpecTrapLimit=0 -XX:StressLongCountedLoop=2000000 -XX:CompileCommand=compileonly,X::* -XX:CompileCommand=printcompilation,X::* X

(rr) p n->dump_bfs(1000,n,"#cA")
dist apd dump
---------------------------------------------
   9  15   99  IfFalse  === 363  [[ 120 ]] #0 !jvms: X::test @ bci:17
   8  11  406  IfFalse  === 387  [[ 405 ]] #0 !orig=[364],99 !jvms: X::test @ bci:17
   8  18  431  IfTrue  === 429  [[ 405 ]] #1 !orig=[404],[364],99 !jvms: X::test @ bci:17
   8  15  120  Loop  === 120 5 99  [[ 120 65 432 ]]  !orig=[26] !jvms: X::test @ bci:2
   7  11  405  Region  === 405 431 406  [[ 405 363 408 ]]  !orig=[364],99 !jvms: X::test @ bci:17
   7  15  432  NeverBranch  === 120  [[ 433 434 ]] 
   6  11  363  If  === 405 412  [[ 99 362 ]] P=0.900000, C=-1.000000
   6  15  433  CProj  === 432  [[ 60 ]] #0
   5  11  362  IfTrue  === 363  [[ 365 ]] #1 !orig=98 !jvms: X::test @ bci:17
   5  15   60  SafePoint  === 433 1 7 1 1 65 1 21 21 21 59  [[ 365 ]]  SafePoint  !jvms: X::test @ bci:17
   4  11  365  Loop  === 365 60 362  [[ 365 388 366 379 397 437 ]] 
   3   4   90  SafePoint  === 386 1 7 1 1 94 1 21 144 144 59  [[ 387 ]]  SafePoint  !jvms: X::test @ bci:17
   3  11  397  SafePoint  === 365 1 7 1 1 388 1 21 389 389 59  [[ 429 ]]  SafePoint  !orig=90 !jvms: X::test @ bci:17
   2   4  387  If  === 90 383  [[ 98 406 ]] P=0.900000, C=-1.000000 !orig=[166],[97] !jvms: X::test @ bci:17
   2  11  429  If  === 397 428  [[ 430 431 ]] P=0.100000, C=-1.000000 !orig=[396],387,[166],[97] !jvms: X::test @ bci:17
   1   4   98  IfTrue  === 387  [[ 386 ]] #1 !jvms: X::test @ bci:17
   1  11  430  IfFalse  === 429  [[ 386 ]] #0 !orig=[395],98 !jvms: X::test @ bci:17
   0   0  386  Loop  === 386 430 98  [[ 386 380 94 90 ]]  !orig=[167],[122],[78] !jvms: X::test @ bci:20

Looks like an infinite-loop bug (NeverBranch), where we do not do things as expected. This seems to happen often with infinite-loops.

Also, these flags seem to be required:

-Xcomp -XX:PerMethodTrapLimit=0 -XX:PerMethodSpecTrapLimit=0 -XX:StressLongCountedLoop=2000000

The effect of these trap limit flags is that there are no alternative exits, hence the infinite-loop. I have seen this before.