JDK-8350840 : C2: x64 Assembler::vpcmpeqq assert: failed: XMM register should be 0-15
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 24,25
  • Priority: P3
  • Status: Open
  • Resolution: Unresolved
  • Submitted: 2025-02-27
  • Updated: 2025-02-28
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 25
25Unresolved
Related Reports
Causes :  
JDK-8338021 - Support new unsigned and saturating vector operators in VectorAPI
Relates :  
JDK-8323582 - C2 SuperWord AlignVector: misaligned vector memory access with unaligned native memory
Sub Tasks
JDK-8350841 :  
ProblemList jdk/incubator/vector/Long256VectorTests.java - Resolved
Description
Seems more likely after JDK-8323582, but I see similar failures on a while back as well, in the valhalla repo. So I don't think that JDK-8323582 is the cause, but it may now make the use of some XMM registers more likely.

Probably only reproduces with AVX512, and possibly only with a subset of those machines (not on my laptop that has only a subset of AVX512 functionality):
jdk/incubator/vector/Long256VectorTests.java

# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/opt/mach5/mesos/work_dir/slaves/03ecc23a-edd5-4bb5-a333-4ff8ea07fd7c-S1547/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/66737b21-4c8d-4877-aa96-f180c9760371/runs/5a6d0c67-5874-4459-9cc4-f30183b0fd30/workspace/open/src/hotspot/cpu/x86/assembler_x86.cpp:13620), pid=2098796, tid=2098815
#  assert(((!is_extended) || (!attributes->is_legacy_mode()))) failed: XMM register should be 0-15
#
# JRE version: Java(TM) SE Runtime Environment (25.0) (fastdebug build 25-internal-LTS-2025-02-27-0711127.emanuel.peter.jdk-fork7)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 25-internal-LTS-2025-02-27-0711127.emanuel.peter.jdk-fork7, mixed mode, sharing, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x66cd78]  Assembler::vex_prefix_and_encode(int, int, int, Assembler::VexSimdPrefix, Assembler::VexOpcode, InstructionAttr*, bool, bool, bool) [clone .constprop.2]+0x2b8
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/03ecc23a-edd5-4bb5-a333-4ff8ea07fd7c-S1941/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/e1d8c66f-7708-45a7-bc8e-d4e603b0dda9/runs/401b2f42-2f9e-4a1f-a29b-cbb74a09981a/testoutput/test-support/jtreg_open_test_jdk_jdk_vector/scratch/0/core.2098796)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp


Current CompileTask:
C2:17089 1722    b        Long256VectorTests::SADDLong256VectorTests (152 bytes)

Stack: [0x00007f42ebafc000,0x00007f42ebbfc000],  sp=0x00007f42ebbf7680,  free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x66cd78]  Assembler::vex_prefix_and_encode(int, int, int, Assembler::VexSimdPrefix, Assembler::VexOpcode, InstructionAttr*, bool, bool, bool) [clone .constprop.2]+0x2b8  (assembler_x86.cpp:13620)
V  [libjvm.so+0x6794dd]  Assembler::vpcmpeqq(XMMRegister, XMMRegister, XMMRegister, int)+0xbd  (assembler_x86.cpp:5104)
V  [libjvm.so+0x8a88a3]  C2_MacroAssembler::vpgenmax_value(BasicType, XMMRegister, XMMRegister, int, bool)+0x33  (c2_MacroAssembler_x86.cpp:6862)
V  [libjvm.so+0x8a905b]  C2_MacroAssembler::vector_addsub_dq_saturating_evex(int, BasicType, XMMRegister, XMMRegister, XMMRegister, XMMRegister, XMMRegister, KRegister, KRegister, int)+0x18b  (c2_MacroAssembler_x86.cpp:6953)
V  [libjvm.so+0x4552a7]  vector_addsub_saturating_reg_evexNode::emit(C2_MacroAssembler*, PhaseRegAlloc*) const+0x2c7  (x86.ad:10685)
V  [libjvm.so+0x159482c]  PhaseOutput::scratch_emit_size(Node const*)+0x3dc  (output.cpp:3385)
V  [libjvm.so+0x158cc2a]  PhaseOutput::shorten_branches(unsigned int*)+0x33a  (output.cpp:539)
V  [libjvm.so+0x159e776]  PhaseOutput::Output()+0xa36  (output.cpp:339)
V  [libjvm.so+0xa6b3da]  Compile::Code_Gen()+0x5aa  (compile.cpp:3017)
V  [libjvm.so+0xa6e18f]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1c9f  (compile.cpp:881)
V  [libjvm.so+0x8ba2e5]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x1d5  (c2compiler.cpp:141)
V  [libjvm.so+0xa7a958]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x928  (compileBroker.cpp:2331)
V  [libjvm.so+0xa7b698]  CompileBroker::compiler_thread_loop()+0x528  (compileBroker.cpp:1975)
V  [libjvm.so+0xf4423e]  JavaThread::thread_main_inner()+0xee  (javaThread.cpp:776)
V  [libjvm.so+0x189b466]  Thread::call_run()+0xb6  (thread.cpp:231)
V  [libjvm.so+0x1573d08]  thread_native_entry(Thread*)+0x128  (os_linux.cpp:877)


Test is randomized, so it might require the seed:

For random generator using seed: 9171110224325618755
To re-run test with same seed value please add "-Djdk.test.lib.random.seed=9171110224325618755" to command line.
Comments
Thanks for taking a look [~sviswanathan]!
28-02-2025
This seems to be due to a problem in vpgenmax_value. In the following code snippet from vpgenmax_value: if (compute_allones) { if (vlen_enc == Assembler::AVX_512bit) { vpternlogd(allones, 0xff, allones, allones, vlen_enc); } else { vpcmpeqq(allones, allones, allones, vlen_enc); } } vpternlogd should be generated for all vlen_enc on avx512vl platforms so the if check needs to be something like: "if ((vlen_enc == Assembler::AVX_512bit) || (VM_Version::supports_avx512vl())) " vpcmpeqq with xmm as destination is a legacy instruction and doesn't support higher bank registers. Similar fix is needed for vpgenmin_value.
27-02-2025
[~sviswanathan], [~jbhateja], could you please have a look at this? Thanks!
27-02-2025
Similar to JDK-8325083. ILW = Assert during C2 compilation, single vector API test, no workaround but disable compilation or AVX = HLM = P3
27-02-2025
Code around 'vector_addsub_saturating_reg_evex' is from JDK-8338021.
27-02-2025