JDK-8273108 : RunThese24H crashes with SEGV in markWord::displaced_mark_helper() after JDK-8268276
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 17,18
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • Submitted: 2021-08-30
  • Updated: 2022-11-11
  • Resolved: 2021-12-12
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 17 JDK 18 JDK 19
17.0.6Fixed 18 b28Fixed 19Fixed
Related Reports
Duplicate :  
JDK-8275427 - RunThese24H.java failed with EXCEPTION_ACCESS_VIOLATION in java.util.ArrayList.add
Relates :  
JDK-8272809 - JFR thread sampler SI_KERNEL SEGV in metaspace::VirtualSpaceList::contains
Relates :  
JDK-8276613 - RunThese30M.java failed with "assert(check_alignment(result)) failed: address not aligned: 0x000000082d2d2d2d"
Relates :  
JDK-8268276 - Base64 Decoding optimization for x86 using AVX-512
Description
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f30dce9d694, pid=17690, tid=22854
#
# JRE version: Java(TM) SE Runtime Environment (18.0+12) (build 18-ea+12-613)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (18-ea+12-613, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xaf2694]  markWord::displaced_mark_helper() const+0x14
#
Comments
A pull request was submitted for review. URL: https://git.openjdk.org/jdk17u-dev/pull/870 Date: 2022-11-10 02:03:18 +0000
10-11-2022
Fix Request 17u Backporting this patch eliminates a bug where decode was overwriting past end of buffer. Low risk as this has been in JDK18 for months. Depends on backport of JDK-8268276, JDK-8269404, JDK-8273459, and JDK-8274527 to cleanly integrate. Patch applies cleanly to 17u. Testing: x86_64 build, affected tests, tier1
08-11-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk17u-dev/pull/864 Date: 2022-11-08 03:35:43 +0000
08-11-2022
The fix for this bug is in jdk-18+28-1965 and jdk-19+2-32.
17-12-2021
Changeset: 9a1bbaf8 Author: Scott Gibbons <scott.gibbons@intel.com> Committer: Vladimir Kozlov <kvn@openjdk.org> Date: 2021-12-12 16:08:35 +0000 URL: https://git.openjdk.java.net/jdk18/commit/9a1bbaf8db0e869ab76be8ab1bd0ddeb23693e7e
12-12-2021
The new AVX512 intrinsic code is also guarded with this: if(VM_Version::supports_avx512_vbmi() && VM_Version::supports_avx512bw()) {
10-12-2021
Note, Base64 intrinsics are only supported on x86 CPU with AVX512: https://github.com/openjdk/jdk/blob/master/src/hotspot/cpu/x86/vm_version_x86.cpp#L967 To verify the fix we need to run on such machine.
10-12-2021
Great job finding this. Thank you. I have submitted a PR and added a test for overrun.
09-12-2021
I believe I have found the bug and am working on a fix. My algorithm was writing 64 bytes when it should only have written 48, which wrote past the end of the array. I will also update the test to catch this issue.
09-12-2021
Very nice analysis [~pchilanomate]! Following your steps, I extracted a simple reproducer which shows the wrong result of decode() (see Reproducer.java): Run: java -Xbatch Reproducer.java Output: Exception in thread "main" java.lang.RuntimeException: Non matching strings! Expected: UmxSU2JGcDZWbGMxVDFac1duTlhiRnBWVmxaS2VWVkdSVGxRVVQwOQ== Found: UmxSU2JGcDZWbGMxVDFac1duTlhiRnBWVmxaS2VWVkdSVGxRVVQwOQ==xxxxxxxx at Reproducer.main(Reproducer.java:39) Hope this helps!
09-12-2021
[~sviswanathan], please help with this.
09-12-2021
ILW = heap corruption with base64 intrinsic, AVX-512 only?, disable intrinsic? = HML = P2
09-12-2021
[~pchilanomate], thanks for tracking this down.
09-12-2021
I think I found the issue. The problem is not in the base64 encoding, it's in the base64 decoding! (JDK-8268276). The 0x7878787878787878 pattern is sometimes being written at the end of these byte[] arrays by code in StubRoutines::_base64_decodeBlock. By looking in gdb at some of the strings that are being decoded it looks like there is no reason for that pattern to be generated and it is just a bug in the decoding implementation. For example, if I add a watchpoint in gdb to catch the moment when 0x7878787878787878 is being written to memory on some byte array I see the following state when hitting the breakpoint: (gdb) info registers rax 0x7f9e02886098 140316624052376 rbx 0x0 0 rcx 0x22f7f7ed0 9386819280 rdx 0x3bcc 15308 rsi 0x0 0 rdi 0x22f7f51d0 9386807760 rbp 0x7f9e19ad8720 0x7f9e19ad8720 rsp 0x7f9e19ad86f0 0x7f9e19ad86f0 r8 0x0 0 r9 0x0 0 r10 0x7f9fa56ebd60 140323652025696 r11 0x0 0 r12 0x0 0 r13 0x0 0 r14 0x4c 76 r15 0x11000 69632 rip 0x7f9fa56ebfa3 0x7f9fa56ebfa3 rip is inside StubRoutines::_base64_decodeBlock: (gdb) p StubRoutines::_base64_decodeBlock $9 = (address) 0x7f9fa56ebd60 (gdb) p StubRoutines::_multiplyToLen (next one to be generated) $10 = (address) 0x7f9fa56ec1a0 rdi points to a byte[] that contains the string we are decoding. Currently it actually points to the last 0x4c bytes (r14) since we are almost done: (gdb) x/14gx 0x22f7f51d0 0x22f7f51d0: 0x7955315534315756 0x587046526a646b53 0x22f7f51e0: 0x455a46654e646b59 0x3152574d6a466d52 0x22f7f51f0: 0x754a56616f784756 0x6868586257646c51 0x22f7f5200: 0x725a3156574a7a55 0x53683352574e465a 0x22f7f5210: 0x52393064525a6c56 0x000000003d305450 0x22f7f5220: 0x0000000000000001 0x000255f800002050 0x22f7f5230: 0x7951326477306d56 0x4f703161576c5855 The header of the byte[] is: (gdb) x/10gx 0x22f7f1640 0x22f7f1640: 0x0000000000000001 0x00003bcc00002050 0x22f7f1650: 0x7951326477306d56 0x58784757566c5855 0x22f7f1660: 0x735a315477465459 0x336c6c525739475a And this is what those last 76 characters that we are decoding look like as a string: (gdb) x/s 0x22f7f51d0 0x22f7f51d0: "VW14U1UySkdjRFpXYkdNeFZERmFjMWR1VGxoaVJuQldWbXhhUzJWV1ZrZFNWR3hSVlZRd09RPT0=" rcx points to the dst byte[] where we are storing the result of the base64 decoding: (gdb) x/12gx 0x22F7F7Ed0 0x22f7f7ed0: 0x474a325553786d55 0x784d4762575a4463 0x22f7f7ee0: 0x7564316361464456 0x57426e5269686c54 0x22f7f7ef0: 0x5756325361786d56 0x5278475653646b56 0x22f7f7f00: 0x3d3d514f77515656 0x7878787878787878 0x22f7f7f10: 0x0000000000000000 0x0000000000000000 0x22f7f7f20: 0x0000000000000000 0x0000000000000000 The header of the byte[] is: (gdb) x/10gx 0x22f7f5220 0x22f7f5220: 0x0000000000000001 0x000255f800002050 0x22f7f5230: 0x7951326477306d56 0x4f703161576c5855 0x22f7f5240: 0x775946566f646c56 0x30786d525852445a And this is what those last characters that we decoded look like as a string: (gdb) x/s 0x22f7f7ed0 0x22f7f7ed0: "UmxSU2JGcDZWbGMxVDFac1duTlhiRnBWVmxaS2VWVkdSVGxRVVQwOQ==xxxxxxxx" The base64 decode of the 76 characters shown initially should be(echo string | base64 -d): "UmxSU2JGcDZWbGMxVDFac1duTlhiRnBWVmxaS2VWVkdSVGxRVVQwOQ==" If we compare it with the string decoded by us we can see the extra xxxxxxxx(0x7878787878787878) generated at the end. If I revert JDK-8268276 and do the same test adding watchpoints I never see that pattern being generated at the end of those byte[]. This base64 test is creating byte arrays of size 11480 for encoding/decoding, and from the cases I've seen the full length of the array is needed for decoding. So, if the base64 decoding stub happens to add these extra bytes then we will overwrite the header of the next object since there is no available space in the array. This would explain the crashes we are seeing. The test also uses some bigger arrays(153080 as we see above) for encoding/decoding but those have a lot of empty space at the end so we wouldn't see any issues there. Also, I've reverted JDK-8268276 and run the test in mach5 800 times already with no failures. Usually with 300 runs on the current tip I get at least 1 crash.
08-12-2021
JDK-8275427 seems to have a corrupted object in eden and this bug seems to also have a corrupted object in eden (copy_to_survivor_space is copying from eden (?)). Therefore, linking them together. JDK-8275427 has a weird pattern too. Different weird pattern tho.
07-12-2021
The theory about the stack overflow exceptions led nowhere. RunThese30.java is failing with no StackOverflowErrors. I looked in the heap at the nearby objects to the one that has the invalid markword (0x7878787878787878) and I see that the previous object contiguously in memory is always a big byte array(~12k bytes) that contains what seems like a base64 encoded string (something like "Vm0wd2QyVkhVWGhVV0dSUFZs.....WVmxaS2VWVkdSVGxRVVQwOQ=="). Interestingly, I found other byte arrays in the heap with the same format, i.e. base64 strings that end with that same special 0x7878787878787878 value. So it seems the 0x7878787878787878 value belongs to the previous byte array that is just overwriting the markword of this object. In other words, this looks a lot like an off-by-one error somewhere, causing an overflow that overwrites the memory of the next contiguous object in memory. This would explain why after adding asserts in all methods that write to the markword to catch the special 0x7878787878787878 value, the test still crashes without hitting the assert. So currently I'm trying to see if I can catch the place where we write to those byte arrays to verify if this is indeed an off-by-one error and try to find where the bug is in the code. I'm also trying to narrow down the commit where the crash first appears but the crash is too intermittent and the test needs to run for about half an hour so it's unreliable and takes a lot of time.
02-12-2021
I've been running RunThese30M.java so far which only runs kitchensink for 30 minutes that's why I wasn't seeing too much. Running RunThese24H.java does show out of memory exceptions during the run, but there is no crash so it doesn't seem to be the reason for the failure we are seeing. I haven't spotted yet stack overflow exceptions, but I only run the test for a couple of hours, not the full 24 hours. I will try to run several instances of RunThese24H.java in parallel the full 24 hours.
18-11-2021
I've been trying to reproduce the crash locally but without success so far. One thing I noticed is that all hs_err files have out of memory and stack overflow exceptions. For example for the jdk-18+21-1265-tier8 crash: OutOfMemory and StackOverflow Exception counts: OutOfMemoryError java_heap_errors=161 StackOverflowErrors=21 LinkageErrors=152172 For the single RunThese30M.java crash spotted there is only one out of memory error: OutOfMemory and StackOverflow Exception counts: OutOfMemoryError java_heap_errors=1 LinkageErrors=12605 I added code to call Exceptions::print_exception_counts_on_error() on VM exit and don't see any OutOfMemory or StackOverflow exceptions in the successful runs. This might indicate the bug is only triggered after an out of memory condition. I tried using -Xmx to reduce the max heap size and see if that triggers an out of memory exception, which in turn might lead to the same crash, but so far the test still passes.
17-11-2021
Here's the stack track for the jdk-18+23-1435-tier3 sighting: applications/runthese/RunThese30M.java --------------- T H R E A D --------------- Current thread (0x00007fb268274ff0): VMThread "VM Thread" [stack: 0x00007fb26d95e000,0x00007fb26da5e000] [id=23517] Stack: [0x00007fb26d95e000,0x00007fb26da5e000], sp=0x00007fb26da5bc70, free space=1015k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x13e1ea0] markWord::displaced_mark_helper() const+0x60 V [libjvm.so+0xa82e20] DefNewGeneration::copy_to_survivor_space(oop)+0x5a0 V [libjvm.so+0xa8c411] void FastScanClosure<DefNewScanClosure>::do_oop_work<narrowOop>(narrowOop*)+0xc1 V [libjvm.so+0xa8c76f] void OopOopIterateDispatch<DefNewScanClosure>::Table::oop_oop_iterate<ObjArrayKlass, narrowOop>(DefNewScanClosure*, oop, Klass*)+0xef V [libjvm.so+0xa881b6] void ContiguousSpace::oop_since_save_marks_iterate<DefNewScanClosure>(DefNewScanClosure*)+0x116 V [libjvm.so+0xa80dc2] DefNewGeneration::FastEvacuateFollowersClosure::do_void()+0x42 V [libjvm.so+0xa81531] DefNewGeneration::collect(bool, bool, unsigned long, bool)+0x581 V [libjvm.so+0xd2e9b7] GenCollectedHeap::collect_generation(Generation*, bool, unsigned long, bool, bool, bool)+0x3e7 V [libjvm.so+0xd2fb8b] GenCollectedHeap::do_collection(bool, bool, unsigned long, bool, GenCollectedHeap::GenerationType)+0x8bb V [libjvm.so+0xd30729] GenCollectedHeap::satisfy_failed_allocation(unsigned long, bool)+0x69 V [libjvm.so+0xd18002] VM_GenCollectForAllocation::doit()+0x52 V [libjvm.so+0x198052a] VM_Operation::evaluate()+0x18a V [libjvm.so+0x19a3ac3] VMThread::evaluate_operation(VM_Operation*)+0x183 V [libjvm.so+0x19a4937] VMThread::inner_execute(VM_Operation*)+0x3f7 V [libjvm.so+0x19a4a65] VMThread::loop()+0xb5 V [libjvm.so+0x19a4b8c] VMThread::run()+0xbc V [libjvm.so+0x1891af0] Thread::call_run()+0x100 V [libjvm.so+0x1572204] thread_native_entry(Thread*)+0x104 siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000
04-11-2021
Here's log file snippets from the jdk-18+18-1044-tier8 sighting: [stress.process.out] [40248.351s][info ][class,unload] unloading class javasoft.sqe.javatest.Test 0x0000000803f7b000 [stress.process.out] [40248.411s][info ][gc ] GC(3586) Pause Remark 535M->535M(1440M) 72.665ms [stress.process.out] [40248.603s][info ][gc ] GC(3586) Pause Cleanup 611M->611M(1440M) 0.218ms [stress.process.out] [40248.628s][info ][gc ] GC(3586) Concurrent Mark Cycle 608.697ms [stress.process.out] [40250.743s][info ][gc ] GC(3587) Pause Young (Prepare Mixed) (G1 Preventive Collection) 1299M->460M(1872M) 16.480ms [stress.process.out] [40250.803s][info ][gc ] GC(3588) Pause Young (Mixed) (G1 Evacuation Pause) 476M->459M(1872M) 18.911ms [stress.process.out] # [stress.process.out] # A fatal error has been detected by the Java Runtime Environment: [stress.process.out] # [stress.process.out] # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff82824fbf1, pid=15124, tid=50484 [stress.process.out] # [stress.process.out] # JRE version: Java(TM) SE Runtime Environment (18.0+18) (build 18-ea+18-1044) [stress.process.out] # Java VM: Java HotSpot(TM) 64-Bit Server VM (18-ea+18-1044, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64) [stress.process.out] # Problematic frame: [stress.process.out] # V [jvm.dll+0x5ffbf1] markWord::displaced_mark_helper+0x21 [stress.process.out] # [stress.process.out] # Core dump will be written. Default location: T:\\testoutput\\test-support\\jtreg_closed_test_hotspot_jtreg_applications_runthese_RunThese24H_java\\scratch\\0\\hs_err_pid15124.mdmp [stress.process.out] # [stress.process.out] Unsupported internal testing APIs have been used. [stress.process.out] [stress.process.out] # An error report file with more information is saved as: [stress.process.out] # T:\\testoutput\\test-support\\jtreg_closed_test_hotspot_jtreg_applications_runthese_RunThese24H_java\\scratch\\0\\hs_err_pid15124.log [stress.process.out] # [stress.process.out] # If you would like to submit a bug report, please visit: [stress.process.out] # https://bugreport.java.com/bugreport/crash.jsp [stress.process.out] # [2021-10-05T21:09:06.353736800Z] Gathering output for process 49436 Here's the crashing thread's stack trace: --------------- T H R E A D --------------- Current thread (0x000001ca697fc9a0): JavaThread "javasoft.sqe.tests.api.java.util.Arrays.AsListStream " daemon [_thread_in_vm, id=50484, stack(0x0000011e53600000,0x0000011e53700000)] Stack: [0x0000011e53600000,0x0000011e53700000], sp=0x0000011e536f9520, free space=997k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0x5ffbf1] markWord::displaced_mark_helper+0x21 (markWord.cpp:45) V [jvm.dll+0x742c51] ObjectSynchronizer::inflate+0x1f1 (synchronizer.cpp:1210) V [jvm.dll+0x74175e] ObjectSynchronizer::FastHashCode+0xfe (synchronizer.cpp:874) V [jvm.dll+0x6609cb] oopDesc::slow_identity_hash+0x8b (oop.cpp:99) V [jvm.dll+0x7b9fdc] Verifier::verify+0xcc (verifier.cpp:172) V [jvm.dll+0x373182] InstanceKlass::link_class_impl+0x2d2 (instanceKlass.cpp:908) V [jvm.dll+0x3722f4] InstanceKlass::initialize_impl+0x44 (instanceKlass.cpp:1071) V [jvm.dll+0x372298] InstanceKlass::initialize+0x28 (instanceKlass.cpp:800) V [jvm.dll+0x381ada] InterpreterRuntime::_new+0x8a (interpreterRuntime.cpp:225) C 0x000001ca077c7877 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j javasoft.sqe.tests.api.java.util.stream.Stream.MapTests.MapTestsBase.map_umbrella(Ljavasoft/sqe/tests/api/java/util/stream/AdjustedStreamData;)V+19 v ~StubRoutines::call_stub J 3717 jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (0 bytes) @ 0x000001ca0f4002a7 [0x000001ca0f400220+0x0000000000000087] J 7424645 c2 jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (137 bytes) @ 0x000001ca0f4fa458 [0x000001ca0f4fa3e0+0x0000000000000078] J 7423783 c2 jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (10 bytes) @ 0x000001ca1169c8d4 [0x000001ca1169c880+0x0000000000000054] J 7437792 c1 com.oracle.tck.lib.autd2.processors.tc.TGFTestCaseMethodSetting.lambda$processSettingMethodPhase$0(Lcom/oracle/tck/lib/autd2/TestCaseContext;)Ljava/lang/Object; (63 bytes) @ 0x000001ca0b19f52c [0x000001ca0b19f2a0+0x000000000000028c] J 7437791 c1 com.oracle.tck.lib.autd2.processors.tc.TGFTestCaseMethodSetting$$Lambda$2208504+0x00000008043f9648.call()Ljava/lang/Object; (12 bytes) @ 0x000001ca09f99094 [0x000001ca09f99020+0x0000000000000074] J 7437781 c1 com.oracle.tck.lib.autd2.processors.tc.DefaultThreadRunning.process(Lcom/oracle/tck/lib/autd2/TestCaseContext$TestCaseLifePhase;Lcom/oracle/tck/lib/autd2/TestCaseContext;)V (48 bytes) @ 0x000001ca09f9a95c [0x000001ca09f9a8e0+0x000000000000007c] J 7437780 c1 com.oracle.tck.lib.autd2.processors.tc.DefaultThreadRunning.process(Lcom/oracle/tck/lib/autd2/LifePhase;Lcom/oracle/tck/lib/autd2/Context;)V (13 bytes) @ 0x000001ca09f9bb9c [0x000001ca09f9bac0+0x00000000000000dc] J 7436615 c1 com.oracle.tck.lib.autd2.AUTD2Utils.iterateThroughProcessorsUntilAllAreDone(Lcom/oracle/tck/lib/autd2/Context;Ljava/util/TreeMap;Lcom/oracle/tck/lib/autd2/LifePhase;Ljava/util/List;)V (207 bytes) @ 0x000001ca0aae8524 [0x000001ca0aae79c0+0x0000000000000b64] J 7456242 c1 com.oracle.tck.lib.autd2.AUTD2Utils.iterateLifePhases(Lcom/oracle/tck/lib/autd2/Context;[Lcom/oracle/tck/lib/autd2/LifePhase;)V (173 bytes) @ 0x000001ca086c9344 [0x000001ca086c8600+0x0000000000000d44] j com.oracle.tck.lib.autd2.AUTD2Utils.iterateTestCaseLifePhase(Lcom/oracle/tck/lib/autd2/TestCaseContext;)Lcom/oracle/tck/lib/autd2/TestResult;+29 j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases.runTestCaseAsNeeded(Lcom/oracle/tck/lib/autd2/TestCaseContext;)Lcom/oracle/tck/lib/autd2/TestResult;+1 j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases.lambda$process$0(Lcom/oracle/tck/lib/autd2/TestGroupContext;Lcom/oracle/tck/lib/autd2/TestCaseContext;)V+24 j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases$$Lambda$2208340+0x000000080365fd30.accept(Ljava/lang/Object;)V+12 J 7425215 c2 java.lang.Iterable.forEach(Ljava/util/function/Consumer;)V java.base@18-ea (39 bytes) @ 0x000001ca119cc6e0 [0x000001ca119cc640+0x00000000000000a0] j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases.process(Lcom/oracle/tck/lib/autd2/TestGroupContext$TestGroupLifePhase;Lcom/oracle/tck/lib/autd2/TestGroupContext;)V+13 j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases.process(Lcom/oracle/tck/lib/autd2/LifePhase;Lcom/oracle/tck/lib/autd2/Context;)V+9 J 7436615 c1 com.oracle.tck.lib.autd2.AUTD2Utils.iterateThroughProcessorsUntilAllAreDone(Lcom/oracle/tck/lib/autd2/Context;Ljava/util/TreeMap;Lcom/oracle/tck/lib/autd2/LifePhase;Ljava/util/List;)V (207 bytes) @ 0x000001ca0aae8524 [0x000001ca0aae79c0+0x0000000000000b64] J 7456242 c1 com.oracle.tck.lib.autd2.AUTD2Utils.iterateLifePhases(Lcom/oracle/tck/lib/autd2/Context;[Lcom/oracle/tck/lib/autd2/LifePhase;)V (173 bytes) @ 0x000001ca086c9344 [0x000001ca086c8600+0x0000000000000d44] j com.oracle.tck.lib.autd2.TestRunner.performRun(Lcom/oracle/tck/lib/autd2/TestGroupContext;Ljava/util/Set;Ljava/util/Set;)V+273 j com.oracle.tck.lib.autd2.TestRunner.run(Ljava/lang/Object;Ljava/io/PrintWriter;Ljava/io/PrintWriter;Ljava/util/Set;Ljava/util/Set;[Ljava/lang/String;)Lcom/oracle/tck/lib/autd2/TestResult;+151 j com.oracle.tck.lib.autd2.AUTD2.run(Ljava/lang/Object;Ljava/io/PrintWriter;Ljava/io/PrintWriter;[Ljava/lang/String;)Ljavasoft/sqe/javatest/Status;+29 j com.oracle.tck.lib.autd2.AUTD2.run([Ljava/lang/String;Ljava/io/PrintWriter;Ljava/io/PrintWriter;)Ljavasoft/sqe/javatest/Status;+4 j javasoft.sqe.tests.api.java.util.stream.GenericStreamTestsBase.runTests(Ljavasoft/sqe/tests/api/java/util/stream/GenericStreamTestsBase$NewTest;)Ljavasoft/sqe/javatest/Status;+26 j javasoft.sqe.tests.api.java.util.stream.GenericStreamTests.checkMapRef1()Ljavasoft/sqe/javatest/Status;+6 v ~StubRoutines::call_stub J 3717 jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (0 bytes) @ 0x000001ca0f4002a7 [0x000001ca0f400220+0x0000000000000087] J 7424645 c2 jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (137 bytes) @ 0x000001ca0f4fa458 [0x000001ca0f4fa3e0+0x0000000000000078] J 7423552 c2 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (65 bytes) @ 0x000001ca11c479f0 [0x000001ca11c47940+0x00000000000000b0] j com.oracle.tck.lib.autd2.processors.tc.DefaultNoArgTestCaseMethodSetting.lambda$process$0(Lcom/oracle/tck/lib/autd2/TestCaseContext;)Ljava/lang/Object;+12 j com.oracle.tck.lib.autd2.processors.tc.DefaultNoArgTestCaseMethodSetting$$Lambda$2208351+0x0000000803acd968.call()Ljava/lang/Object;+4 J 7437781 c1 com.oracle.tck.lib.autd2.processors.tc.DefaultThreadRunning.process(Lcom/oracle/tck/lib/autd2/TestCaseContext$TestCaseLifePhase;Lcom/oracle/tck/lib/autd2/TestCaseContext;)V (48 bytes) @ 0x000001ca09f9a95c [0x000001ca09f9a8e0+0x000000000000007c] J 7437780 c1 com.oracle.tck.lib.autd2.processors.tc.DefaultThreadRunning.process(Lcom/oracle/tck/lib/autd2/LifePhase;Lcom/oracle/tck/lib/autd2/Context;)V (13 bytes) @ 0x000001ca09f9bb9c [0x000001ca09f9bac0+0x00000000000000dc] J 7436615 c1 com.oracle.tck.lib.autd2.AUTD2Utils.iterateThroughProcessorsUntilAllAreDone(Lcom/oracle/tck/lib/autd2/Context;Ljava/util/TreeMap;Lcom/oracle/tck/lib/autd2/LifePhase;Ljava/util/List;)V (207 bytes) @ 0x000001ca0aae8524 [0x000001ca0aae79c0+0x0000000000000b64] J 7456242 c1 com.oracle.tck.lib.autd2.AUTD2Utils.iterateLifePhases(Lcom/oracle/tck/lib/autd2/Context;[Lcom/oracle/tck/lib/autd2/LifePhase;)V (173 bytes) @ 0x000001ca086c9344 [0x000001ca086c8600+0x0000000000000d44] j com.oracle.tck.lib.autd2.AUTD2Utils.iterateTestCaseLifePhase(Lcom/oracle/tck/lib/autd2/TestCaseContext;)Lcom/oracle/tck/lib/autd2/TestResult;+29 j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases.runTestCaseAsNeeded(Lcom/oracle/tck/lib/autd2/TestCaseContext;)Lcom/oracle/tck/lib/autd2/TestResult;+1 j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases.lambda$process$0(Lcom/oracle/tck/lib/autd2/TestGroupContext;Lcom/oracle/tck/lib/autd2/TestCaseContext;)V+24 j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases$$Lambda$2208340+0x000000080365fd30.accept(Ljava/lang/Object;)V+12 J 7425215 c2 java.lang.Iterable.forEach(Ljava/util/function/Consumer;)V java.base@18-ea (39 bytes) @ 0x000001ca119cc6e0 [0x000001ca119cc640+0x00000000000000a0] j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases.process(Lcom/oracle/tck/lib/autd2/TestGroupContext$TestGroupLifePhase;Lcom/oracle/tck/lib/autd2/TestGroupContext;)V+13 j com.oracle.tck.lib.autd2.processors.tg.RunningTestCases.process(Lcom/oracle/tck/lib/autd2/LifePhase;Lcom/oracle/tck/lib/autd2/Context;)V+9 j com.oracle.tck.lib.autd2.AUTD2Utils.iterateThroughProcessorsUntilAllAreDone(Lcom/oracle/tck/lib/autd2/Context;Ljava/util/TreeMap;Lcom/oracle/tck/lib/autd2/LifePhase;Ljava/util/List;)V+120 j com.oracle.tck.lib.autd2.AUTD2Utils.iterateLifePhases(Lcom/oracle/tck/lib/autd2/Context;[Lcom/oracle/tck/lib/autd2/LifePhase;)V+72 j com.oracle.tck.lib.autd2.TestRunner.performRun(Lcom/oracle/tck/lib/autd2/TestGroupContext;Ljava/util/Set;Ljava/util/Set;)V+273 j com.oracle.tck.lib.autd2.TestRunner.run(Ljava/lang/Object;Ljava/io/PrintWriter;Ljava/io/PrintWriter;Ljava/util/Set;Ljava/util/Set;[Ljava/lang/String;)Lcom/oracle/tck/lib/autd2/TestResult;+151 j com.oracle.tck.lib.autd2.AUTD2.run(Ljava/lang/Object;Ljava/io/PrintWriter;Ljava/io/PrintWriter;[Ljava/lang/String;)Ljavasoft/sqe/javatest/Status;+29 j com.oracle.tck.lib.autd2.AUTD2.run(Ljava/lang/Object;[Ljava/lang/String;)Ljavasoft/sqe/javatest/Status;+34 j javasoft.sqe.tests.api.java.util.Arrays.AsListStream.main([Ljava/lang/String;)V+8 v ~StubRoutines::call_stub J 3717 jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (0 bytes) @ 0x000001ca0f4002a7 [0x000001ca0f400220+0x0000000000000087] J 7424645 c2 jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (137 bytes) @ 0x000001ca0f4fa458 [0x000001ca0f4fa3e0+0x0000000000000078] J 7423783 c2 jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (10 bytes) @ 0x000001ca1169c8d4 [0x000001ca1169c880+0x0000000000000054] J 7428361 c1 applications.kitchensink.process.stress.modules.JckStressModule$TestRunner$1.run()V (127 bytes) @ 0x000001ca0ab1598c [0x000001ca0ab15800+0x000000000000018c] v ~StubRoutines::call_stub siginfo: EXCEPTION_ACCESS_VIOLATION (0xc0000005), reading address 0xffffffffffffffff
05-10-2021
(gdb) print old $11 = {_o = 0x2a2aab478} (gdb) x /g 0x2a2aab478 0x2a2aab478: 0x7878787878787878 0x2a2aab480: 0x0000000400001240 0x2a2aab488: 0xffe0f8554821afba 0x2a2aab490: 0x4821972effe0f847 The old_mark passed in is also 0x7878 etc. But the klass looks ok. (gdb) print *klass $5 = {<Metadata> = {<MetaspaceObj> = {}, _vptr.Metadata = 0x800000728, _valid = 0}, _layout_helper = -2146431998, _id = ObjArrayKlassID, _vtable_len = 5, _super_check_offset = 64, _name = 0x800490a80, _secondary_super_cache = 0x800001678, _secondary_supers = 0x80060f200, _primary_supers = {0x800000e18, 0x800001240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, _java_mirror = {_obj = 0x7f2eb01309d0}, _super = 0x800000e18, _subklass = 0x0, _next_sibling = 0x800002840, _next_link = 0x800000e18, _class_loader_data = 0x7f2eb01145d0, _modifier_flags = 1041, _access_flags = {_flags = -2113929216}, _trace_id = 13898241, _shared_class_path_index = -1, _shared_class_flags = 0, _archived_mirror_index = -1} (gdb) print *klass->_next_link $6 = {<Metadata> = {<MetaspaceObj> = {}, _vptr.Metadata = 0x8000000d0, _valid = 0}, _layout_helper = 16, _id = InstanceKlassID, _vtable_len = 5, _super_check_offset = 56, _name = 0x80048a068, _secondary_super_cache = 0x0, _secondary_supers = 0x80060f218, _primary_supers = {0x800000e18, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, _java_mirror = { _obj = 0x7f2eb01309c8}, _super = 0x0, _subklass = 0x80128f7c8, _next_sibling = 0x0, _next_link = 0x0, _class_loader_data = 0x7f2eb01145d0, _modifier_flags = 1, _access_flags = {_flags = 589299745}, _trace_id = 13834757, _shared_class_path_index = 0, _shared_class_flags = 8, _archived_mirror_index = -1}
16-09-2021
But now also seen on Windows, so not an SI_KERNEL issue.
15-09-2021
The SI_KERNEL here is very suspect. There is another bug that also shows SI_KERNEL SEGVs - JDK-8272809.
01-09-2021
Here's the crashing thread's stack for the jdk-18+12-613-tier8 sighting: --------------- T H R E A D --------------- Current thread (0x00007f2fe8656e00): JavaThread "javasoft.sqe.tests.api.javax.print.PrintService.hashCodeTests " daemon [_thread_in_vm, id=22854, stack(0x00007f2ea5dfe000,0x00007f2ea5eff000)] Stack: [0x00007f2ea5dfe000,0x00007f2ea5eff000], sp=0x00007f2ea5efc4e8, free space=1017k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xaf2694] markWord::displaced_mark_helper() const+0x14 V [libjvm.so+0xced941] ObjectSynchronizer::enter(Handle, BasicLock*, JavaThread*)+0xc1 V [libjvm.so+0xc57acb] SharedRuntime::complete_monitor_locking_C(oopDesc*, BasicLock*, JavaThread*)+0x6b Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) v ~RuntimeStub::_complete_monitor_locking_Java J 1272960 c2 java.util.ConcurrentModificationException.<init>()V java.base@18-ea (5 bytes) @ 0x00007f30c87a36f8 [0x00007f30c87a32c0+0x0000000000000438] J 1273319% c2 javax.print.PrintServiceLookup.lookupDefaultPrintService()Ljavax/print/PrintService; java.desktop@18-ea (46 bytes) @ 0x00007f30c5521828 [0x00007f30c5521760+0x00000000000000c8] j javasoft.sqe.tests.api.javax.print.PrintService.hashCodeTests.hashCode001()Ljavasoft/sqe/javatest/Status;+0 v ~StubRoutines::call_stub J 2610 jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (0 bytes) @ 0x00007f30c53976bf [0x00007f30c5397640+0x000000000000007f] J 1147161 c2 jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (137 bytes) @ 0x00007f30c9803758 [0x00007f30c98036e0+0x0000000000000078] J 1145644 c2 jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (10 bytes) @ 0x00007f30c5407ef4 [0x00007f30c5407ea0+0x0000000000000054] J 1145801 c2 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (65 bytes) @ 0x00007f30c56fe084 [0x00007f30c56fe000+0x0000000000000084] j javasoft.sqe.javatest.lib.MultiTest.invokeTestCase(Ljava/lang/reflect/Method;)Ljavasoft/sqe/javatest/Status;+8 j javasoft.sqe.javatest.lib.MultiTest.run([Ljava/lang/String;Ljava/io/PrintWriter;Ljava/io/PrintWriter;)Ljavasoft/sqe/javatest/Status;+139 j javasoft.sqe.javatest.lib.MultiTest.run([Ljava/lang/String;Ljava/io/PrintStream;Ljava/io/PrintStream;)Ljavasoft/sqe/javatest/Status;+40 j javasoft.sqe.tests.api.javax.print.PrintService.hashCodeTests.main([Ljava/lang/String;)V+16 v ~StubRoutines::call_stub J 2610 jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (0 bytes) @ 0x00007f30c53976bf [0x00007f30c5397640+0x000000000000007f] J 1147161 c2 jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (137 bytes) @ 0x00007f30c9803758 [0x00007f30c98036e0+0x0000000000000078] J 1145644 c2 jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; java.base@18-ea (10 bytes) @ 0x00007f30c5407ef4 [0x00007f30c5407ea0+0x0000000000000054] J 1168081 c1 applications.kitchensink.process.stress.modules.JckStressModule$TestRunner$1.run()V (127 bytes) @ 0x00007f30c364fc1c [0x00007f30c364faa0+0x000000000000017c] v ~StubRoutines::call_stub siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000
31-08-2021
Here's the crashing thread's stack for the jdk-18+12-580-tier8 sighting: --------------- T H R E A D --------------- Current thread (0x00007f2e480ca810): GCTaskThread "GC Thread#6" [stack: 0x00007f2dda963000,0x00007f2ddaa63000] [id=11445] Stack: [0x00007f2dda963000,0x00007f2ddaa63000], sp=0x00007f2ddaa61980, free space=1018k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x13d19b0] markWord::displaced_mark_helper() const+0x60 V [libjvm.so+0xcac6ec] G1ParScanThreadState::do_copy_to_survivor_space(G1HeapRegionAttr, oop, markWord)+0x2cc V [libjvm.so+0xcb18e0] void G1ParScanThreadState::do_oop_evac<narrowOop>(narrowOop*)+0xd0 V [libjvm.so+0xcad6f3] G1ParScanThreadState::trim_queue_to_threshold(unsigned int)+0x343 V [libjvm.so+0xcada07] G1ParScanThreadState::steal_and_trim_queue(GenericTaskQueueSet<OverflowTaskQueue<ScannerTask, (MEMFLAGS)5, 131072u>, (MEMFLAGS)5>*)+0x277 V [libjvm.so+0xc22de7] G1ParEvacuateFollowersClosure::do_void()+0x157 V [libjvm.so+0xc2c1a8] G1EvacuateRegionsBaseTask::evacuate_live_objects(G1ParScanThreadState*, unsigned int, G1GCPhaseTimes::GCParPhases, G1GCPhaseTimes::GCParPhases)+0x98 V [libjvm.so+0xc24d36] G1EvacuateRegionsBaseTask::work(unsigned int)+0x66 V [libjvm.so+0x19e13a5] GangWorker::run_task(WorkData)+0x85 V [libjvm.so+0x19e14e4] GangWorker::loop()+0x44 V [libjvm.so+0x19e153a] V [libjvm.so+0x1885e80] Thread::call_run()+0x100 V [libjvm.so+0x1565284] thread_native_entry(Thread*)+0x104 siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000
31-08-2021
ILW = HLM = p3
31-08-2021