==================
Problem
==================

When testing jtreg cases under "test/hotspot/jtreg/compiler, test/hotspot/jtreg/gc", we got many ZGC related test failures on Thunder X CPU and Thunder X2 CPU.

==================
Here lists the failed test cases.
==================

compiler/gcbarriers/TestZGCBarrierElision.java#id0
compiler/gcbarriers/TestZGCBarrierElision.java#id1
compiler/gcbarriers/UnsafeIntrinsicsTest.java#ZGenerationalDebug
compiler/loopopts/TestRangeCheckPredicatesControl.java#ZGenerational
compiler/loopstripmining/TestNoWarningLoopStripMiningIterSet.java#ZGenerational
compiler/uncommontrap/TestDeoptOOM.java#ZGenerational
compiler/vectorapi/VectorRebracket128Test.java#ZGenerational 
gc/TestReferenceClearDuringReferenceProcessing.java#ZGenerational
gc/TestSystemGC.java#ZGenerational
gc/stress/gcbasher/TestGCBasherWithZ.java#id0
gc/stress/gcbasher/TestGCBasherWithZ.java#id2
gc/stress/gcold/TestGCOldWithZ.java#id0
gc/stringdedup/TestStringDeduplicationAgeThreshold.java#ZGenerational
gc/stringdedup/TestStringDeduplicationFullGC.java#ZGenerational
gc/stringdedup/TestStringDeduplicationInterned.java#ZGenerational
gc/stringdedup/TestStringDeduplicationPrintOptions.java#ZGenerational
gc/stringdedup/TestStringDeduplicationTableResize.java#ZGenerational
gc/stringdedup/TestStringDeduplicationYoungGC.java#ZGenerational
gc/z/TestAllocateHeapAt.java
gc/z/TestAlwaysPreTouch.java
gc/z/TestGarbageCollectorMXBean.java
gc/z/TestMemoryMXBean.java
gc/z/TestMemoryManagerMXBean.java
gc/z/TestNoUncommit.java
gc/z/TestPageCacheFlush.java
gc/z/TestRelocateInPlace.java
gc/z/TestSmallHeap.java
gc/z/TestUncommit.java
gc/z/TestZForceDiscontiguousHeapReservations.java
gc/z/TestZNMT.java 

==================
Error message:
==================

----------System.out:(17/813)----------
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (~/jdk_build/jdk_src/src/hotspot/share/asm/codeBuffer.hpp:200), pid=108369, tid=108373
#  assert(allocates2(pc)) failed: not in CodeBuffer memory: 0x0000ffff9c921100 <= 0x0000ffff9c934c04 <= 0x0000ffff9c934c00
#
# JRE version:  (22.0) (fastdebug build )
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 22-internal-git-0916e6a60, mixed mode, sharing, compressed class ptrs, z gc, linux-aarch64)
# Problematic frame:
# V  [libjvm.so+0x45413c]  Instruction_aarch64::~Instruction_aarch64()+0xbc
#
# Core dump will be written. Default location: /tmp/core.108369

==================
Backtrace info
==================

Stack: [0x0000ffffada02000,0x0000ffffadc00000],  sp=0x0000ffffadbfcb50,  free space=2026k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x4542bc]  Instruction_aarch64::~Instruction_aarch64()+0xbc  (codeBuffer.hpp:200)
V  [libjvm.so+0x121a154]  MacroAssembler::pop(unsigned int, Register)+0x174  (assembler_aarch64.hpp:1489)
V  [libjvm.so+0x195e300]  ZCopyRuntimeCallSpill::restore()+0x3a0  (macroAssembler_aarch64.hpp:465)
V  [libjvm.so+0x19575b0]  copy_load_barrier(MacroAssembler*, Register, Address, Register)+0x120  (zBarrierSetAssembler_aarch64.cpp:438)
V  [libjvm.so+0x1957ab4]  copy_load_barrier(MacroAssembler*, FloatRegister, Address, Register, Register, FloatRegister)+0x3e4  (zBarrierSetAssembler_aarch64.cpp:511)
V  [libjvm.so+0x195cd24]  ZBarrierSetAssembler::copy_load_at(MacroAssembler*, unsigned long, BasicType, unsigned long, FloatRegister, FloatRegister, Address, Register, Register, FloatRegister)+0x234  (zBarrierSetAssembler_aarch64.cpp:734)
V  [libjvm.so+0x16ccf6c]  StubGenerator::ArrayCopyBarrierSetHelper::copy_load_at_32(FloatRegister, FloatRegister, Address)+0xac  (stubGenerator_aarch64.cpp:736)
V  [libjvm.so+0x16e42f4]  StubGenerator::copy_memory(unsigned long, BasicType, bool, Register, Register, Register, int)+0x684  (stubGenerator_aarch64.cpp:1241)
V  [libjvm.so+0x16e5ab0]  StubGenerator::generate_conjoint_copy(int, bool, bool, unsigned char*, unsigned char**, char const*, bool)+0x320  (stubGenerator_aarch64.cpp:1579)
V  [libjvm.so+0x16ed7c8]  StubGenerator::generate_arraycopy_stubs()+0x448  (stubGenerator_aarch64.cpp:1810)
V  [libjvm.so+0x16c8c40]  StubGenerator_generate(CodeBuffer*, StubCodeGenerator::StubsKind)+0x53c  (stubGenerator_aarch64.cpp:8292)
V  [libjvm.so+0x16f304c]  initialize_stubs(StubCodeGenerator::StubsKind, int, int, char const*, char const*, char const*)+0x11c  (stubRoutines.cpp:234)
V  [libjvm.so+0x16f41ec]  StubRoutines::initialize_final_stubs()+0x56c  (stubRoutines.cpp:318)
V  [libjvm.so+0xd671b0]  init_globals2()+0x70  (init.cpp:180)
V  [libjvm.so+0x17ac1c8]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x318  (threads.cpp:564)
V  [libjvm.so+0xec5ecc]  JNI_CreateJavaVM+0x7c  (jni.cpp:3577)
C  [libjli.so+0x3ebc]  JavaMain+0x7c  (java.c:1506)
C  [libjli.so+0x73bc]  ThreadJavaMain+0xc  (java_md.c:650)
C  [libc.so.6+0x7d5c8] 


==================
Some notes
==================
1. we didn't see the failures in other CPUs, including Neoverse N1 and N2.

2. the failure can also be reproduced in Thunder X and X2 via the following test command

`./jdk/bin/java -XX:+UseZGC -XX:+ZGenerational --version `

3. from the backtrace, we can see, it's an assembler failure and the VM failed during generating "final stubs".

4. based on my investigation, the root cause should be that VM flag "AvoidUnalignedAccesses" is enabled by default on CPUs like Thunder X and X2. Hence, more instructions would be generated especially with ZGC. As a result, the buffer size for final stubs overflows.

This failure should be also reproduced on all(or almost) AArch64 CPUs with the following command:

`./jdk/bin/java -XX:+UseZGC -XX:+ZGenerational -XX:+AvoidUnalignedAccesses --version`

Note that I verified this on Neoverse N1 and N2.