Bug ID: JDK-8291830 jvmti/RedefineClasses/StressRedefine failed: assert(!is_null(v)) failed: narrow klass value can never be zero

JDK-8291830 : jvmti/RedefineClasses/StressRedefine failed: assert(!is_null(v)) failed: narrow klass value can never be zero

Type: Bug
Component: hotspot
Sub-Component: jvmti
Affected Version: 19

Priority: P2
Status: Resolved
Resolution: Fixed
OS: windows
CPU: x86_64

Submitted: 2022-08-03
Updated: 2023-09-28
Resolved: 2023-09-28

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 11	JDK 17	JDK 20
11.0.21-oracleFixed	17.0.9-oracleFixed	20 b27Fixed

Related Reports

Relates :	JDK-8249822 - SymbolPropertyTable creates an extra OopHandle per entry
Relates :	JDK-8296915 - assert(Universe::is_in_heap(result)) failed: object not in heap 0x0000000000000010
Relates :	JDK-8293584 - CodeCache::old_nmethods_do incorrectly filters is_unloading nmethods

Description

The following test failed in the JDK19 CI:

vmTestbase/nsk/jvmti/RedefineClasses/StressRedefine/TestDescription.java

Here's a snippet from the log file:

Got expected exception: java.lang.reflect.InvocationTargetException
Got expected exception: java.lang.reflect.InvocationTargetException
Got expected exception: java.lang.reflect.InvocationTargetException
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=\\oops/compressedOops.inline.hpp:133
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (c:\\sb\\prod\\1659451653\\workspace\\open\\src\\hotspot\\share\\oops/compressedOops.inline.hpp:133), pid=32764, tid=35388
#  assert(!is_null(v)) failed: narrow klass value can never be zero
#
# JRE version: Java(TM) SE Runtime Environment (19.0+34) (fastdebug build 19-ea+34-2227)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 19-ea+34-2227, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, windows-amd64)
# Core dump will be written. Default location: C:\\sb\\prod\\1659511560\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_quick\\scratch\\3\\hs_err_pid32764.mdmp
#
# An error report file with more information is saved as:
# C:\\sb\\prod\\1659511560\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_quick\\scratch\\3\\hs_err_pid32764.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
ee the JVMTI spec.
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
<<<<<<<< RedefineClasses() is successfully done
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
<<<<<<<< RedefineClasses() is successfully done
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
c:\\sb\\prod\\1659451653\\workspace\\open\\test\\hotspot\\jtreg\\vmTestbase\\nsk\\jvmti\\RedefineClasses\\StressRedefine\\stressRedefine.cpp: Failed to call RedefineClasses():
	the function returned error 62: JVMTI_ERROR_FAILS_VERIFICATION
	For more info about this error see the JVMTI spec.
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
c:\\sb\\prod\\1659451653\\workspace\\open\\test\\hotspot\\jtreg\\vmTestbase\\nsk\\jvmti\\RedefineClasses\\StressRedefine\\stressRedefine.cpp: Failed to call RedefineClasses():
	the function returned error 60: JVMTI_ERROR_INVALID_CLASS_FORMAT
	For more info about this error see the JVMTI spec.
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
<<<<<<<< RedefineClasses() is successfully done
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
c:\\sb\\prod\\1659451653\\workspace\\open\\test\\hotspot\\jtreg\\vmTestbase\\nsk\\jvmti\\RedefineClasses\\StressRedefine\\stressRedefine.cpp: Failed to call RedefineClasses():
	the function returned error 67: JVMTI_ERROR_UNSUPPORTED_REDEFINITION_METHOD_DELETED
	For more info about this error see the JVMTI spec.
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
<<<<<<<< RedefineClasses() is successfully done
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
c:\\sb\\prod\\1659451653\\workspace\\open\\test\\hotspot\\jtreg\\vmTestbase\\nsk\\jvmti\\RedefineClasses\\StressRedefine\\stressRedefine.cpp: Failed to call RedefineClasses():
	the function returned error 60: JVMTI_ERROR_INVALID_CLASS_FORMAT
	For more info about this error see the JVMTI spec.
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
<<<<<<<< RedefineClasses() is successfully done
c:\\sb\\prod\\1659451653\\workspace\\open\\test\\hotspot\\jtreg\\vmTestbase\\nsk\\jvmti\\RedefineClasses\\StressRedefine\\stressRedefine.cpp: Failed to call RedefineClasses():
	the function returned error 62: JVMTI_ERROR_FAILS_VERIFICATION
	For more info about this error see the JVMTI spec.
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
c:\\sb\\prod\\1659451653\\workspace\\open\\test\\hotspot\\jtreg\\vmTestbase\\nsk\\jvmti\\RedefineClasses\\StressRedefine\\stressRedefine.cpp: Failed to call RedefineClasses():
	the function returned error 62: JVMTI_ERROR_FAILS_VERIFICATION
	For more info about this error see the JVMTI spec.
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
c:\\sb\\prod\\1659451653\\workspace\\open\\test\\hotspot\\jtreg\\vmTestbase\\nsk\\jvmti\\RedefineClasses\\StressRedefine\\stressRedefine.cpp: Failed to call RedefineClasses():
	the function returned error 62: JVMTI_ERROR_FAILS_VERIFICATION
	For more info about this error see the JVMTI spec.
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2353
----------System.err:(11/3643)*----------


Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x000001bff51b5c00):  VMThread "VM Thread" [stack: 0x0000003db8000000,0x0000003db8100000] [id=35388]

Stack: [0x0000003db8000000,0x0000003db8100000]
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0xc853f1]  os::platform_print_native_stack+0xf1  (os_windows_x86.cpp:235)
V  [jvm.dll+0xee78eb]  VMError::report+0x10eb  (vmError.cpp:835)
V  [jvm.dll+0xee942e]  VMError::report_and_die+0x7fe  (vmError.cpp:1687)
V  [jvm.dll+0xee9bb4]  VMError::report_and_die+0x64  (vmError.cpp:1468)
V  [jvm.dll+0x5a72a7]  report_vm_error+0xb7  (debug.cpp:283)
V  [jvm.dll+0x22af2]  oopDesc::klass+0x52  (oop.inline.hpp:86)
V  [jvm.dll+0x812c16]  java_lang_Class::set_klass+0x76  (javaClasses.cpp:1508)
V  [jvm.dll+0x7c5faf]  InstanceKlass::deallocate_contents+0x6f  (instanceKlass.cpp:583)
V  [jvm.dll+0x5022f1]  ClassLoaderData::free_deallocate_list+0x381  (classLoaderData.cpp:866)
V  [jvm.dll+0x505d6d]  ClassLoaderDataGraph::clean_deallocate_lists+0x7d  (classLoaderDataGraph.cpp:164)
V  [jvm.dll+0xa5232d]  VM_RedefineClasses::doit+0x1ed  (jvmtiRedefineClasses.cpp:298)
V  [jvm.dll+0xef0927]  VM_Operation::evaluate+0xc7  (vmOperations.cpp:71)
V  [jvm.dll+0xef2575]  VMThread::evaluate_operation+0xb5  (vmThread.cpp:283)
V  [jvm.dll+0xef2d26]  VMThread::inner_execute+0x256  (vmThread.cpp:432)
V  [jvm.dll+0xef30a4]  VMThread::run+0x154  (vmThread.cpp:175)
V  [jvm.dll+0xe59e0c]  Thread::call_run+0x1ac  (thread.cpp:366)
V  [jvm.dll+0xc83c99]  thread_native_entry+0xb9  (os_windows.cpp:545)
C  [ucrtbase.dll+0x1fb80]
C  [KERNEL32.DLL+0x84d4]
C  [ntdll.dll+0x51791]

VM_Operation (0x0000003dbb7feff0): RedefineClasses, mode: safepoint, requested by thread 0x000001bffbbee7c0, redefining class MyClass

There's a much older unresolved bug that mentions the same assert:

JDK-8203350 Crash in vmTestbase/nsk/jvmti/scenarios/hotswap/HS201/hs201t002/TestDescription.java

Comments

Fix request [11u] I backport this for parity with 11.0.21-oracle. The change does not at all apply cleanly. See PR for details. SAP nightly tests did not reveal any issue related to this PR.
04-07-2023
A pull request was submitted for review. URL: https://git.openjdk.org/jdk11u-dev/pull/2019 Date: 2023-07-03 15:41:39 +0000
03-07-2023
Fix Request (17u): Should get backported for parity with 17.0.9-oracle. Applies cleanly. The related issues don't seem to be caused by this change and are not backported atm.
20-06-2023
A pull request was submitted for review. URL: https://git.openjdk.org/jdk17u-dev/pull/1478 Date: 2023-06-20 15:52:53 +0000
20-06-2023
Changeset: fb6fd032 Author: Coleen Phillimore <coleenp@openjdk.org> Date: 2022-12-02 19:09:05 +0000 URL: https://git.openjdk.org/jdk/commit/fb6fd03233b0eb001e2995d20a079b6af31d2b9b
02-12-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/11444 Date: 2022-12-01 02:01:54 +0000
01-12-2022
I know why this failed. The oop that we replace in the OopHandle is in a Handle so should be safe, but the ClassLoaderData has an optimization that sets modified_oops to avoid CLDG walking and the replace for RedefineClases doesn't set that.
01-12-2022
For class file verification, RedefineClasses replaces the mirror in the scratch class with the mirror in the class we are redefining because of some issue with verification. This code should copy the OopHandle, and not replace the oop that the OopHandle points to. It seems like when we do this GC misses the oop (even though we save it in a Handle). This was broken in JDK 16 with JDK-8249822.
01-12-2022
This same failure happens with JDK 19. Edit: I see it was logged against JDK 19. I was trying to see if it was a regression in JDK 20 (so, not a regression).
30-11-2022
It's trying to redefine MyClass over and over with various broken classfiles. It is the same class that is being updated. I'm confused why is the mirror zero though? Actually this is a good clue. We poke the scratch class mirror into the real class to run the verifier (class file verifier). I'll look at that code.
30-11-2022
[~coleenp]: I did some tracking of the mirror of one of the klasses and would like to confirm whether what I'm seeing makes sense to you: Every GC is represented by two gc,verify log lines in my changes, one for before-gc verification, one for after-gc verification. Every log line shows the klass address and the mirror address. [10.751s][debug][gc,verify] GC(3) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 [10.856s][debug][gc,verify] GC(3) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 Redefinition VM Op happens here [11.677s][debug][gc,verify] GC(4) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 [11.785s][debug][gc,verify] GC(4) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 [12.443s][debug][gc,verify] GC(5) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 [12.547s][debug][gc,verify] GC(5) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 [13.194s][debug][gc,verify] GC(6) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 [13.323s][debug][gc,verify] GC(6) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 [13.964s][debug][gc,verify] GC(7) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 [14.140s][debug][gc,verify] GC(7) k: 0x0000000801166c00 MyClass mirror 0x0000000000000000 [... mutator poking mirror...] [14.846s][debug][gc,verify] GC(8) k: 0x0000000801166c00 MyClass mirror 0x00000000c00122d0 (old gen) [14.991s][debug][gc,verify] GC(8) k: 0x0000000801166c00 MyClass mirror 0x00000000c00122d0 (old gen) [15.636s][debug][gc,verify] GC(9) k: 0x0000000801166c00 MyClass mirror 0x00000000c00122d0 (old gen) [15.777s][debug][gc,verify] GC(9) k: 0x0000000801166c00 MyClass mirror 0x00000000c00122d0 (old gen) [... mutator poking mirror again...] [16.721s][debug][gc,verify] GC(10) k: 0x0000000801166c00 MyClass mirror 0x00000000f52b8098 (to-space??) [16.906s][debug][gc,verify] GC(10) k: 0x0000000801166c00 MyClass mirror 0x00000000f52b8098 (from-space??) So it looks like some mutator code (class redefinition) sets the mirror twice for this klass. Since there is no klass unloading going on due to zero full gcs, I believe that this is always the same klass that is being updated. Can you confirm that what I'm seeing with logging is reasonable from an execution order and steps taken during class redefinition (in some case)? Note that in GC(10), what is poked by the mutator in is an invalid, outdated pointer in to-space(!). Existing verification (for performance reasons) only checks whether a reference is in committed areas. From the hs_err file I can't see a reason for to-space being occupied (that would immediately trigger a full gc which never happens). So actually, this is a before-gc verification error, making this smell like a naked oop in jvmti/class redefintion. Can you check again - this is really a code area out of my expertise and probably you could find that much quicker.
30-11-2022
Fwiw, if there ever had been doubt, the broken mirror is from a "MyClass" klass that is being redefined (verified by some additional logging).
30-11-2022
There are more bad oops in CLDs with G1 also. See linked bug. These all seem related.
29-11-2022
I added VerifyBeforeGC/VerifyAfterGC and My crash looks like yours. It's a VerifyAfterGC where the mirror is broken. V [jvm.dll+0xc7d54f] PSScavenge::invoke_no_policy+0x163f (psScavenge.cpp:664) <== VerifyAfterGC V [jvm.dll+0x185bf] oopDesc::klass+0x9f (oop.inline.hpp:86) V [jvm.dll+0xa20688] Klass::verify_on+0x1b8 (klass.cpp:791) V [jvm.dll+0x7a19cb] InstanceKlass::verify_on+0x4b (instanceKlass.cpp:3738) V [jvm.dll+0x4d58f7] ClassLoaderData::verify+0x1b7 (classLoaderData.cpp:1043) V [jvm.dll+0x4d7541] ClassLoaderDataGraph::verify+0xc1 (classLoaderDataGraph.cpp:649) V [jvm.dll+0xe2c9b8] Universe::verify+0x2d8 (universe.cpp:1155) V [jvm.dll+0xc7d54f] PSScavenge::invoke_no_policy+0x163f (psScavenge.cpp:664) V [jvm.dll+0xc7be57] PSScavenge::invoke+0x187 (psScavenge.cpp:243) V [jvm.dll+0xc18246] ParallelScavengeHeap::failed_mem_allocate+0x146 (parallelScavengeHeap.cpp:470) V [jvm.dll+0xc7e0ac] VM_ParallelGCFailedAllocation::doit+0x4c (psVMOperations.cpp:45) V [jvm.dll+0xe8cbe8] VM_Operation::evaluate+0xc8 (vmOperations.cpp:72) I changed Klass::verify_on() to have a more specific test so I got the deafbabe is unaligned crash. if (java_mirror_no_keepalive() != NULL) { guarantee(java_lang_Class::is_instance(java_mirror_no_keepalive()), "should be instance"); } I also added more verification to ClassLoaderData::verify() (verifying InstanceKlasses in deallocate lists, and an oops_do for the OopHandles). This is assigned to GC so I'm going to take my name off because I can't do anything further. I did verify that the deallocate lists are working like I think they should work. When we create a class when parsing, we added it to the ClassLoaderData::_klasses list almost immediately. When the mirror is created, it's added to the _handle list and to the Klass. If that fails, it's null. I don't see a possibility for partial initialization in this respect. Windows debugging doesn't work for me otherwise, I'd love to know if the whole CLD klasses are bad and why it only happens on windows.
29-11-2022
I've been running this and it only fails on Windows. I got a new symptom of : # assert(PSScavenge::should_scavenge(p, true)) failed: revisiting object? Also walking the CLDG handle list.
29-11-2022
Running with -XX:+VerifyBeforeGC and -XX:+VerifyAfterGC, the VM fails during after-gc verification with the following error/stack trace: # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (klass.cpp:791), pid=45952, tid=33520 # guarantee(oopDesc::is_oop(java_mirror_no_keepalive())) failed: should be instance # [...] Current thread (0x000002e3545fdbb0): VMThread "VM Thread" [stack: 0x000000f62aa00000,0x000000f62ab00000] [id=26096] Stack: [0x000000f62aa00000,0x000000f62ab00000] Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0x6ba2ca] os::win32::platform_print_native_stack+0xca (os_windows_x86.cpp:236) V [jvm.dll+0x84aeb9] VMError::report+0xc19 (vmError.cpp:814) V [jvm.dll+0x84c92e] VMError::report_and_die+0x5fe (vmError.cpp:1572) V [jvm.dll+0x84cfd7] VMError::report_and_die+0x47 (vmError.cpp:1353) V [jvm.dll+0x280d5a] report_vm_error+0x8a (debug.cpp:286) V [jvm.dll+0x59b01b] Klass::verify_on+0x14b (klass.cpp:791) V [jvm.dll+0x3ccf1a] InstanceKlass::verify_on+0x2a (instanceKlass.cpp:3738) V [jvm.dll+0x227210] ClassLoaderData::verify+0xf0 (classLoaderData.cpp:1027) V [jvm.dll+0x2287c7] ClassLoaderDataGraph::verify+0xa7 (classLoaderDataGraph.cpp:648) V [jvm.dll+0x814b15] Universe::verify+0x2d5 (universe.cpp:1155) V [jvm.dll+0x6fa666] PSScavenge::invoke_no_policy+0xe26 (psScavenge.cpp:664) V [jvm.dll+0x6f9784] PSScavenge::invoke+0x34 (psScavenge.cpp:243) V [jvm.dll+0x6c5813] ParallelScavengeHeap::failed_mem_allocate+0x33 (parallelScavengeHeap.cpp:470) So it seems that we are missing some roots somewhere (as this is a young/scavenge gc which does not unload classes) in case of (probably) unsuccessful class redefinition.
29-11-2022
Just verification, still crashing with JDK-8297499 and JDK-8293584 resolved.
29-11-2022
Not sure about what "there are no classes that get unloaded with this test" means, but given JDK-8297499 classes might get unloaded forcibly ;) It might be worth rechecking with that fix in.
23-11-2022
There might be a timing hole between failed class loading in redefinition and creating the mirror, assigning to myself. Edit. probably not. cld_do() isn't deallocating the class because I don't think this class loader is ever unloaded, it should be walking the oops/mirrors in the classes and keeping them alive. Yes, there are no classes that get unloaded with this test.
04-11-2022
I actually think there is something wrong with ParallelGC because of this and JDK-8294744 and there have been changes in parallel fairly recently. I'll see if I can get anymore info out of these crashes before passing it back to gc/Unassigned.
03-11-2022
Some details on one of the asserts: The hs_err dump is: # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (c:\sb\prod\1666691125\workspace\open\src\hotspot\share\oops/compressedOops.inline.hpp:135), pid=128744, tid=11628 # assert(check_alignment(result)) failed: address not aligned: 0x00000008deafbabe # # JRE version: Java(TM) SE Runtime Environment (20.0) (fastdebug build 20-internal-2022-10-25-0943383.serguei.spitsyn.jdk20) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-internal-2022-10-25-0943383.serguei.spitsyn.jdk20, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, windows-amd64) # Core dump will be written. Default location: C:\sb\prod\1666692652\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\scratch\0\hs_err_pid128744.mdmp # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # --------------- S U M M A R Y ------------ Command Line: -XX:MaxRAMPercentage=4.16667 -Djava.io.tmpdir=c:\sb\prod\1666692652\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\tmp -XX:+CreateCoredumpOnCrash -XX:+UseParallelGC -XX:+UseNUMA -Djava.library.path=c:\ade\mesos\work_dir\jib-master\install\2022-10-25-0943383.serguei.spitsyn.jdk20\windows-x64-debug.test\hotspot\jtreg\native -agentlib:stressRedefine nsk.jvmti.RedefineClasses.StressRedefine ./bin Host: win2016-x64-683836, AMD EPYC 7J13 64-Core Processor , 12 cores, 23G, Windows Server 2016 , 64 bit Build 14393 (10.0.14393.3630) Time: Tue Oct 25 11:11:44 2022 /GM elapsed time: 13.948147 seconds (0d 0h 0m 13s) --------------- T H R E A D --------------- Current thread (0x0000022bbb62f390): VMThread "VM Thread" [stack: 0x00000089ec800000,0x00000089ec900000] [id=11628] Stack: [0x00000089ec800000,0x00000089ec900000] Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xc084a1] os::win32::platform_print_native_stack+0xf1 (os_windows_x86.cpp:236) V [jvm.dll+0xe6ccce] VMError::report+0x10be (vmError.cpp:841) V [jvm.dll+0xe6e7c4] VMError::report_and_die+0x644 (vmError.cpp:1700) V [jvm.dll+0xe6ef04] VMError::report_and_die+0x64 (vmError.cpp:1481) V [jvm.dll+0x57c8d7] report_vm_error+0xb7 (debug.cpp:285) V [jvm.dll+0x18dcf] oopDesc::klass+0x9f (oop.inline.hpp:86) V [jvm.dll+0x7e78d6] java_lang_Class::set_klass+0x76 (javaClasses.cpp:1416) V [jvm.dll+0x797dee] InstanceKlass::deallocate_contents+0xae (instanceKlass.cpp:609) V [jvm.dll+0x4dbafd] ClassLoaderData::free_deallocate_list+0x3bd (classLoaderData.cpp:861) V [jvm.dll+0x4df22f] ClassLoaderDataGraph::clean_deallocate_lists+0x10f (classLoaderDataGraph.cpp:180) V [jvm.dll+0xa08ae5] VM_RedefineClasses::doit+0x1f5 (jvmtiRedefineClasses.cpp:304) V [jvm.dll+0xe76058] VM_Operation::evaluate+0xc8 (vmOperations.cpp:72) V [jvm.dll+0xe77c3c] VMThread::evaluate_operation+0x9c (vmThread.cpp:282) V [jvm.dll+0xe78376] VMThread::inner_execute+0x256 (vmThread.cpp:431) V [jvm.dll+0xe786e0] VMThread::run+0x150 (vmThread.cpp:175) V [jvm.dll+0xde7077] Thread::call_run+0x257 (thread.cpp:229) V [jvm.dll+0xc06db8] thread_native_entry+0xb8 (os_windows.cpp:547) C [ucrtbase.dll+0x1fb80] C [KERNEL32.DLL+0x84d4] C [ntdll.dll+0x51791] VM_Operation (0x00000089eedfec80): RedefineClasses, mode: safepoint, requested by thread 0x0000022bc305f860, redefining class MyClass The InstanceKlass being deallocated is the tested class MyClass. These are the corresponding tracing lines: DBG: CLDG::cld_do: thr: 0000022BC36A3F00 ld: 0000022BC2601AF0 . . . . . . . . . DBG: CLD::free_deallocate_list: free_metadata thr: 0000022BBB62F390 ld: 0000022BC2601AF0 ik: 000000080115D000 name: MyClass DBG: IK::deallocate_contents: thr: 0000022BBB62F390 ld: 0000022BC2601AF0 ik: 000000080115D000 name: MyClass So, it seems the same class loader data "ld: 0000022BC2601AF0" is concurrently deallocated by ClassLoaderDataGraph::cld_do() and ClassLoaderData::free_deallocate_list(). The matching mach5 link of the test result with tracing is: https://mach5.us.oracle.com/mdash/jobs/sspitsyn-nsk-jvmti-StressRedefine-20221025-0945-37775426/results?search=status%3Afailed%20AND%20-state%3Ainvalid
01-11-2022
This issue is reproducible with the -XX:+UseParallelGC only. I was not able to reproduce it in 1000 runs with: -XX:+UseSerialGC, -XX:+UseG1GC and -XX:+UseZGC. With -XX:+UseParallelGC it is normally reproducible in 400 runs on windows-x64.
27-10-2022
Coleen, how important is the -XX:+UseParallelGC for us? Should it be a priority?
27-10-2022
I can't imagine ParallelGC having changed to cause this but we should reassign this bug to GC to evaluate. It could just be a timing thing but it's odd.
27-10-2022
Yes. I've already figured it out. It generates bad bytecodes with the probability 75%. This issue is not reproducible without the option -XX:+UseParallelGC. The option -XX:+UseNUMA is not really needed to reproduce this issue.
26-10-2022
> It looks like this test is not healthy and allows JVMTI RedefineClasses to silently fail. This StressRedefine test case randomly flips bits in the new classfiles so that it is supposed to give errors during the redefinition. It's on purpose.
25-10-2022
I'll check if the problems are still reproducible without UseParallelGC. Also, I'm not sure the use of -XX:+UseNUMA is necessary for reproducibility.
22-10-2022
The odd thing is that all these failures are UseParallelGC and so is JDK-8294744. JDK-8294744 has redefinition but it fails in InterpreterRuntime::exception_handler_for_exception on linux-aarch64-debug.
21-10-2022
I've checked if this issue can be a dup of JDK-8293584. My observation is that it is not a dup but it is somehow related. Please, see the following comment with the temporary fix and hs_err dumps.
20-10-2022
I've applied the following fix to check if this bug is a dup of JDK-8293584: diff --git a/src/hotspot/share/code/codeCache.cpp b/src/hotspot/share/code/codeCache.cpp index 40bc3bf602a..d883af9a9e4 100644 --- a/src/hotspot/share/code/codeCache.cpp +++ b/src/hotspot/share/code/codeCache.cpp @@ -1308,9 +1308,7 @@ void CodeCache::old_nmethods_do(MetadataClosure* f) { for (int i = 0; i < length; i++) { CompiledMethod* cm = old_compiled_method_table->at(i); // Only walk !is_unloading nmethods, the other ones will get removed by the GC. - if (!cm->is_unloading()) { - old_compiled_method_table->at(i)->metadata_do(f); - } + old_compiled_method_table->at(i)->metadata_do(f); } } log_debug(redefine, class, nmethod)("Walked %d nmethods for mark_on_stack", length); The results is that the original assert is still reproducible. But two new assert kind of crash modes were discovered (see below). ============================================================ # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (c:\sb\prod\1666178691\workspace\open\src\hotspot\share\classfile\javaClasses.cpp:1474), pid=57604, tid=17392 # assert(is_instance(java_class)) failed: must be a Class object # # JRE version: Java(TM) SE Runtime Environment (20.0) (fastdebug build 20-internal-2022-10-19-1112210.serguei.spitsyn.jdk20) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-internal-2022-10-19-1112210.serguei.spitsyn.jdk20, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, windows-amd64) # Core dump will be written. Default location: C:\sb\prod\1666188271\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\scratch\0\hs_err_pid57604.mdmp # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # --------------- S U M M A R Y ------------ Command Line: -XX:MaxRAMPercentage=4.16667 -Djava.io.tmpdir=c:\sb\prod\1666188271\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\tmp -XX:+CreateCoredumpOnCrash -XX:+UseParallelGC -XX:+UseNUMA -Djava.library.path=c:\ade\mesos\work_dir\jib-master\install\2022-10-19-1112210.serguei.spitsyn.jdk20\windows-x64-debug.test\hotspot\jtreg\native -agentlib:stressRedefine nsk.jvmti.RedefineClasses.StressRedefine ./bin Host: win2016-x64-401196, AMD EPYC 7J13 64-Core Processor , 12 cores, 23G, Windows Server 2016 , 64 bit Build 14393 (10.0.14393.3630) Time: Wed Oct 19 15:05:21 2022 /GM elapsed time: 11.400167 seconds (0d 0h 0m 11s) --------------- T H R E A D --------------- Current thread (0x000001992ddfea30): VMThread "VM Thread" [stack: 0x00000040d7300000,0x00000040d7400000] [id=17392] Stack: [0x00000040d7300000,0x00000040d7400000] Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xc090e1] os::win32::platform_print_native_stack+0xf1 (os_windows_x86.cpp:236) V [jvm.dll+0xe6b99e] VMError::report+0x10be (vmError.cpp:841) V [jvm.dll+0xe6d494] VMError::report_and_die+0x644 (vmError.cpp:1700) V [jvm.dll+0xe6dbd4] VMError::report_and_die+0x64 (vmError.cpp:1481) V [jvm.dll+0x57c5a7] report_vm_error+0xb7 (debug.cpp:285) V [jvm.dll+0x7e77aa] java_lang_Class::set_klass+0xba (javaClasses.cpp:1474) V [jvm.dll+0x797cdf] InstanceKlass::deallocate_contents+0x6f (instanceKlass.cpp:584) V [jvm.dll+0x4dbc01] ClassLoaderData::free_deallocate_list+0x381 (classLoaderData.cpp:860) V [jvm.dll+0x4df0dd] ClassLoaderDataGraph::clean_deallocate_lists+0x7d (classLoaderDataGraph.cpp:173) V [jvm.dll+0xa08705] VM_RedefineClasses::doit+0x1f5 (jvmtiRedefineClasses.cpp:304) V [jvm.dll+0xe74d08] VM_Operation::evaluate+0xc8 (vmOperations.cpp:72) V [jvm.dll+0xe768ec] VMThread::evaluate_operation+0x9c (vmThread.cpp:282) V [jvm.dll+0xe77026] VMThread::inner_execute+0x256 (vmThread.cpp:431) V [jvm.dll+0xe77390] VMThread::run+0x150 (vmThread.cpp:175) V [jvm.dll+0xde65f7] Thread::call_run+0x257 (thread.cpp:229) V [jvm.dll+0xc079f8] thread_native_entry+0xb8 (os_windows.cpp:547) C [ucrtbase.dll+0x1fb80] C [KERNEL32.DLL+0x84d4] C [ntdll.dll+0x51791] VM_Operation (0x00000040db2ff220): RedefineClasses, mode: safepoint, requested by thread 0x000001994f8d78a0, redefining class MyClass ============================================================== # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (c:\sb\prod\1666178691\workspace\open\src\hotspot\share\gc/parallel/psClosure.inline.hpp:94), pid=19092, tid=8140 # assert(PSScavenge::should_scavenge(p, true)) failed: revisiting object? # # JRE version: Java(TM) SE Runtime Environment (20.0) (fastdebug build 20-internal-2022-10-19-1112210.serguei.spitsyn.jdk20) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-internal-2022-10-19-1112210.serguei.spitsyn.jdk20, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, windows-amd64) # Core dump will be written. Default location: C:\sb\prod\1666187694\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\scratch\0\hs_err_pid19092.mdmp # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # --------------- S U M M A R Y ------------ Command Line: -XX:MaxRAMPercentage=4.16667 -Djava.io.tmpdir=c:\sb\prod\1666187694\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\tmp -XX:+CreateCoredumpOnCrash -XX:+UseParallelGC -XX:+UseNUMA -Djava.library.path=c:\ade\mesos\work_dir\jib-master\install\2022-10-19-1112210.serguei.spitsyn.jdk20\windows-x64-debug.test\hotspot\jtreg\native -agentlib:stressRedefine nsk.jvmti.RedefineClasses.StressRedefine ./bin Host: win2016-x64-970318, AMD EPYC 7J13 64-Core Processor , 12 cores, 23G, Windows Server 2016 , 64 bit Build 14393 (10.0.14393.3630) Time: Wed Oct 19 14:55:52 2022 /GM elapsed time: 17.756780 seconds (0d 0h 0m 17s) --------------- T H R E A D --------------- Current thread (0x000001ce179f2e10): WorkerThread "GC Thread#4" [stack: 0x0000003c1c300000,0x0000003c1c400000] [id=8140] Stack: [0x0000003c1c300000,0x0000003c1c400000] Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xc090e1] os::win32::platform_print_native_stack+0xf1 (os_windows_x86.cpp:236) V [jvm.dll+0xe6b99e] VMError::report+0x10be (vmError.cpp:841) V [jvm.dll+0xe6d494] VMError::report_and_die+0x644 (vmError.cpp:1700) V [jvm.dll+0xe6dbd4] VMError::report_and_die+0x64 (vmError.cpp:1481) V [jvm.dll+0x57c5a7] report_vm_error+0xb7 (debug.cpp:285) V [jvm.dll+0xc8010c] PSScavengeFromCLDClosure::do_oop+0x1cc (psClosure.inline.hpp:94) V [jvm.dll+0x4dd1ec] ClassLoaderData::ChunkedHandleList::oops_do_chunk+0x4c (classLoaderData.cpp:213) V [jvm.dll+0x4dd08c] ClassLoaderData::ChunkedHandleList::oops_do+0x3c (classLoaderData.cpp:225) V [jvm.dll+0xc7f98a] PSScavengeCLDClosure::do_cld+0x6a (psClosure.inline.hpp:136) V [jvm.dll+0x4df04b] ClassLoaderDataGraph::cld_do+0x3b (classLoaderDataGraph.cpp:278) V [jvm.dll+0xc827ae] ScavengeRootsTask::work+0x23e (psScavenge.cpp:330) V [jvm.dll+0xeae7b7] WorkerThread::run+0x97 (workerThread.cpp:164) V [jvm.dll+0xde65f7] Thread::call_run+0x257 (thread.cpp:229) V [jvm.dll+0xc079f8] thread_native_entry+0xb8 (os_windows.cpp:547) C [ucrtbase.dll+0x1fb80] C [KERNEL32.DLL+0x84d4] C [ntdll.dll+0x51791]
19-10-2022
This issue is reproducible with a little bit different asserts on Windows: # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (c:\sb\prod\1665797520\workspace\open\src\hotspot\share\oops/compressedOops.inline.hpp:135), pid=29756, tid=23788 # assert(check_alignment(result)) failed: address not aligned: 0x00000008deafbabe # # JRE version: Java(TM) SE Runtime Environment (20.0) (fastdebug build 20-internal-2022-10-15-0130374.serguei.spitsyn.jdk20.1) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-internal-2022-10-15-0130374.serguei.spitsyn.jdk20.1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, windows-amd64) # Core dump will be written. Default location: C:\sb\prod\1666172024\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\scratch\0\hs_err_pid29756.mdmp # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # --------------- S U M M A R Y ------------ Command Line: -XX:MaxRAMPercentage=4.16667 -Djava.io.tmpdir=c:\sb\prod\1666172024\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\tmp -XX:+CreateCoredumpOnCrash -XX:+UseParallelGC -XX:+UseNUMA -Djava.library.path=c:\ade\mesos\work_dir\jib-master\install\2022-10-15-0130374.serguei.spitsyn.jdk20.1\windows-x64-debug.test\hotspot\jtreg\native -agentlib:stressRedefine nsk.jvmti.RedefineClasses.StressRedefine ./bin Host: win2016-x64-169294, AMD EPYC 7J13 64-Core Processor , 12 cores, 23G, Windows Server 2016 , 64 bit Build 14393 (10.0.14393.3630) Time: Wed Oct 19 10:34:36 2022 /GM elapsed time: 14.256130 seconds (0d 0h 0m 14s) --------------- T H R E A D --------------- Current thread (0x000002927aa6e960): VMThread "VM Thread" [stack: 0x000000bc06200000,0x000000bc06300000] [id=23788] Stack: [0x000000bc06200000,0x000000bc06300000] Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xc3ad41] os::win32::platform_print_native_stack+0xf1 (os_windows_x86.cpp:236) V [jvm.dll+0xe8db97] VMError::report+0x10e7 (vmError.cpp:838) V [jvm.dll+0xe8f72e] VMError::report_and_die+0x65e (vmError.cpp:1686) V [jvm.dll+0xe8fea4] VMError::report_and_die+0x64 (vmError.cpp:1467) V [jvm.dll+0x583027] report_vm_error+0xb7 (debug.cpp:284) V [jvm.dll+0x18eaf] oopDesc::klass+0x9f (oop.inline.hpp:86) V [jvm.dll+0x7eeaf6] java_lang_Class::set_klass+0x76 (javaClasses.cpp:1472) V [jvm.dll+0x79eeff] InstanceKlass::deallocate_contents+0x6f (instanceKlass.cpp:585) V [jvm.dll+0x4e29d1] ClassLoaderData::free_deallocate_list+0x381 (classLoaderData.cpp:854) V [jvm.dll+0x4e5e7d] ClassLoaderDataGraph::clean_deallocate_lists+0x7d (classLoaderDataGraph.cpp:164) V [jvm.dll+0xa0e165] VM_RedefineClasses::doit+0x1f5 (jvmtiRedefineClasses.cpp:304) V [jvm.dll+0xe96f2d] VM_Operation::evaluate+0xcd (vmOperations.cpp:72) V [jvm.dll+0xe98b26] VMThread::evaluate_operation+0xa6 (vmThread.cpp:283) V [jvm.dll+0xe992c6] VMThread::inner_execute+0x256 (vmThread.cpp:432) V [jvm.dll+0xe99630] VMThread::run+0x150 (vmThread.cpp:175) V [jvm.dll+0xe079b7] Thread::call_run+0x257 (thread.cpp:229) V [jvm.dll+0xc39688] thread_native_entry+0xb8 (os_windows.cpp:547) C [ucrtbase.dll+0x1fb80] C [KERNEL32.DLL+0x84d4] C [ntdll.dll+0x51791] VM_Operation (0x000000bc087ff040): RedefineClasses, mode: safepoint, requested by thread 0x000002921ce44f40, redefining class MyClass --------------- P R O C E S S --------------- Threads class SMR info: _java_thread_list=0x000002921bf25bf0, length=73, elements={ 0x000002927844ff20, 0x000002927aa7dab0, 0x000002927b920f00, 0x000002927b9234a0, 0x000002927b924310, 0x000002927b925d40, 0x000002927b9278c0, 0x000002927b935980, 0x000002927b938f90, 0x000002927bb6cb30, 0x000002927bba1810, 0x000002921bba57e0, 0x000002921c291300, 0x000002921c7d69f0, 0x000002921caff4d0, 0x000002921caa32d0, 0x000002921cf330a0, 0x000002921cf33680, 0x000002921c7183b0, 0x000002921c718990, 0x000002921c403c50, 0x000002921c735050, 0x000002921c733810, 0x000002921c731fd0, 0x000002921c7325e0, 0x000002921c733e20, 0x000002921c732bf0, 0x000002921c734430, 0x000002921c733200, 0x000002921c7319c0, 0x000002921c734a40, 0x000002921ce3f450, 0x000002921ce3d600, 0x000002921ce412a0, 0x000002921ce44f40, 0x000002921ce43d10, 0x000002921ce418b0, 0x000002921ce43700, 0x000002921ce40680, 0x000002921ce3e830, 0x000002921ce44320, 0x000002921ce42ae0, 0x000002921ce3dc10, 0x000002921ce41ec0, 0x000002921ce424d0, 0x000002921ce40c90, 0x000002921ce430f0, 0x000002921ce44930, 0x000002921ce3e220, 0x000002921ce3ee40, 0x000002921ce3fa60, 0x000002921ce40070, 0x000002921ce8ea20, 0x000002921ce8f030, 0x000002921ce8cbd0, 0x000002921ce8ad80, 0x000002921ce8bfb0, 0x000002921ce8fc50, 0x000002921ce8c5c0, 0x000002921ce8de00, 0x000002921ce8d1e0, 0x000002921ce8d7f0, 0x000002921ce8e410, 0x000002921ce8a160, 0x000002921ce91490, 0x000002921ce8f640, 0x000002921ce90260, 0x000002921ce8b390, 0x000002921ce8b9a0, 0x000002921ce90870, 0x000002921ce91aa0, 0x000002921ce8a770, 0x000002921ce90e80 } _java_thread_list_alloc_cnt=78, _java_thread_list_free_cnt=76, _java_thread_list_max=73, _nested_thread_list_max=0 _tlh_cnt=2682, _tlh_times=1227, avg_tlh_time=0.46, _tlh_time_max=17 _deleted_thread_cnt=2, _deleted_thread_times=0, avg_deleted_thread_time=0.00, _deleted_thread_time_max=0 _delete_lock_wait_cnt=0, _delete_lock_wait_max=0 _to_delete_list_cnt=0, _to_delete_list_max=2 Java Threads: ( => current thread ) 0x000002927844ff20 JavaThread "main" [_thread_blocked, id=21100, stack(0x000000bc06000000,0x000000bc06100000)] 0x000002927aa7dab0 JavaThread "Reference Handler" daemon [_thread_blocked, id=58312, stack(0x000000bc06300000,0x000000bc06400000)] 0x000002927b920f00 JavaThread "Finalizer" daemon [_thread_blocked, id=26504, stack(0x000000bc06400000,0x000000bc06500000)] 0x000002927b9234a0 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=51492, stack(0x000000bc06500000,0x000000bc06600000)] 0x000002927b924310 JavaThread "Attach Listener" daemon [_thread_blocked, id=14644, stack(0x000000bc06600000,0x000000bc06700000)] 0x000002927b925d40 JavaThread "Service Thread" daemon [_thread_blocked, id=25256, stack(0x000000bc06700000,0x000000bc06800000)] 0x000002927b9278c0 JavaThread "Monitor Deflation Thread" daemon [_thread_blocked, id=34844, stack(0x000000bc06800000,0x000000bc06900000)] 0x000002927b935980 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=15720, stack(0x000000bc06900000,0x000000bc06a00000)] 0x000002927b938f90 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=25832, stack(0x000000bc06a00000,0x000000bc06b00000)] 0x000002927bb6cb30 JavaThread "Notification Thread" daemon [_thread_blocked, id=36708, stack(0x000000bc06b00000,0x000000bc06c00000)] 0x000002927bba1810 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=32720, stack(0x000000bc06d00000,0x000000bc06e00000)] 0x000002921bba57e0 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=31772, stack(0x000000bc07000000,0x000000bc07100000)] 0x000002921c291300 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=37780, stack(0x000000bc07100000,0x000000bc07200000)] 0x000002921c7d69f0 JavaThread "Thread-0" [_thread_blocked, id=43688, stack(0x000000bc07200000,0x000000bc07300000)] 0x000002921caff4d0 JavaThread "Thread-1" [_thread_blocked, id=7748, stack(0x000000bc07300000,0x000000bc07400000)] 0x000002921caa32d0 JavaThread "Thread-2" [_thread_blocked, id=50660, stack(0x000000bc07400000,0x000000bc07500000)] 0x000002921cf330a0 JavaThread "Thread-3" [_thread_blocked, id=20604, stack(0x000000bc07500000,0x000000bc07600000)] 0x000002921cf33680 JavaThread "Thread-4" [_thread_blocked, id=12628, stack(0x000000bc07600000,0x000000bc07700000)] 0x000002921c7183b0 JavaThread "Thread-5" [_thread_blocked, id=10332, stack(0x000000bc07700000,0x000000bc07800000)] 0x000002921c718990 JavaThread "Thread-6" [_thread_blocked, id=21580, stack(0x000000bc07800000,0x000000bc07900000)] 0x000002921c403c50 JavaThread "Thread-7" [_thread_blocked, id=48600, stack(0x000000bc07900000,0x000000bc07a00000)] 0x000002921c735050 JavaThread "Thread-8" [_thread_blocked, id=57392, stack(0x000000bc07a00000,0x000000bc07b00000)] 0x000002921c733810 JavaThread "Thread-9" [_thread_blocked, id=17356, stack(0x000000bc07b00000,0x000000bc07c00000)] 0x000002921c731fd0 JavaThread "Thread-10" [_thread_blocked, id=47704, stack(0x000000bc07c00000,0x000000bc07d00000)] 0x000002921c7325e0 JavaThread "Thread-11" [_thread_blocked, id=42748, stack(0x000000bc07d00000,0x000000bc07e00000)] 0x000002921c733e20 JavaThread "Thread-12" [_thread_blocked, id=35352, stack(0x000000bc07e00000,0x000000bc07f00000)] 0x000002921c732bf0 JavaThread "Thread-13" [_thread_blocked, id=8432, stack(0x000000bc07f00000,0x000000bc08000000)] 0x000002921c734430 JavaThread "Thread-14" [_thread_blocked, id=15460, stack(0x000000bc08000000,0x000000bc08100000)] 0x000002921c733200 JavaThread "Thread-15" [_thread_blocked, id=53332, stack(0x000000bc08100000,0x000000bc08200000)] 0x000002921c7319c0 JavaThread "Thread-16" [_thread_blocked, id=25608, stack(0x000000bc08200000,0x000000bc08300000)] 0x000002921c734a40 JavaThread "Thread-17" [_thread_blocked, id=51356, stack(0x000000bc08300000,0x000000bc08400000)] 0x000002921ce3f450 JavaThread "Thread-18" [_thread_blocked, id=3692, stack(0x000000bc08400000,0x000000bc08500000)] 0x000002921ce3d600 JavaThread "Thread-19" [_thread_blocked, id=21380, stack(0x000000bc08500000,0x000000bc08600000)] 0x000002921ce412a0 JavaThread "Thread-20" [_thread_blocked, id=35300, stack(0x000000bc08600000,0x000000bc08700000)] 0x000002921ce44f40 JavaThread "Thread-21" [_thread_blocked, id=30564, stack(0x000000bc08700000,0x000000bc08800000)] 0x000002921ce43d10 JavaThread "Thread-22" [_thread_blocked, id=21060, stack(0x000000bc08800000,0x000000bc08900000)] 0x000002921ce418b0 JavaThread "Thread-23" [_thread_blocked, id=18556, stack(0x000000bc08900000,0x000000bc08a00000)] 0x000002921ce43700 JavaThread "Thread-24" [_thread_blocked, id=34800, stack(0x000000bc08a00000,0x000000bc08b00000)] 0x000002921ce40680 JavaThread "Thread-25" [_thread_blocked, id=48240, stack(0x000000bc08b00000,0x000000bc08c00000)] 0x000002921ce3e830 JavaThread "Thread-26" [_thread_blocked, id=47312, stack(0x000000bc08c00000,0x000000bc08d00000)] 0x000002921ce44320 JavaThread "Thread-27" [_thread_blocked, id=50956, stack(0x000000bc08d00000,0x000000bc08e00000)] 0x000002921ce42ae0 JavaThread "Thread-28" [_thread_blocked, id=18484, stack(0x000000bc08e00000,0x000000bc08f00000)] 0x000002921ce3dc10 JavaThread "Thread-29" [_thread_blocked, id=34568, stack(0x000000bc08f00000,0x000000bc09000000)] 0x000002921ce41ec0 JavaThread "Thread-30" [_thread_blocked, id=36576, stack(0x000000bc09000000,0x000000bc09100000)] 0x000002921ce424d0 JavaThread "Thread-31" [_thread_blocked, id=19404, stack(0x000000bc09100000,0x000000bc09200000)] 0x000002921ce40c90 JavaThread "Thread-32" [_thread_blocked, id=28276, stack(0x000000bc09200000,0x000000bc09300000)] 0x000002921ce430f0 JavaThread "Thread-33" [_thread_blocked, id=20636, stack(0x000000bc09300000,0x000000bc09400000)] 0x000002921ce44930 JavaThread "Thread-34" [_thread_blocked, id=22132, stack(0x000000bc09400000,0x000000bc09500000)] 0x000002921ce3e220 JavaThread "Thread-35" [_thread_blocked, id=9884, stack(0x000000bc09500000,0x000000bc09600000)] 0x000002921ce3ee40 JavaThread "Thread-36" [_thread_blocked, id=10132, stack(0x000000bc09f00000,0x000000bc0a000000)] 0x000002921ce3fa60 JavaThread "Thread-37" [_thread_blocked, id=58224, stack(0x000000bc0a000000,0x000000bc0a100000)] 0x000002921ce40070 JavaThread "Thread-38" [_thread_blocked, id=13140, stack(0x000000bc0a100000,0x000000bc0a200000)] 0x000002921ce8ea20 JavaThread "Thread-39" [_thread_blocked, id=57368, stack(0x000000bc0a200000,0x000000bc0a300000)] 0x000002921ce8f030 JavaThread "Thread-40" [_thread_blocked, id=24464, stack(0x000000bc0a300000,0x000000bc0a400000)] 0x000002921ce8cbd0 JavaThread "Thread-41" [_thread_blocked, id=35392, stack(0x000000bc0a400000,0x000000bc0a500000)] 0x000002921ce8ad80 JavaThread "Thread-42" [_thread_blocked, id=30728, stack(0x000000bc0a500000,0x000000bc0a600000)] 0x000002921ce8bfb0 JavaThread "Thread-43" [_thread_blocked, id=32604, stack(0x000000bc0a600000,0x000000bc0a700000)] 0x000002921ce8fc50 JavaThread "Thread-44" [_thread_blocked, id=27280, stack(0x000000bc0a700000,0x000000bc0a800000)] 0x000002921ce8c5c0 JavaThread "Thread-45" [_thread_blocked, id=47656, stack(0x000000bc0a800000,0x000000bc0a900000)] 0x000002921ce8de00 JavaThread "Thread-46" [_thread_blocked, id=16436, stack(0x000000bc0a900000,0x000000bc0aa00000)] 0x000002921ce8d1e0 JavaThread "Thread-47" [_thread_blocked, id=32940, stack(0x000000bc0aa00000,0x000000bc0ab00000)] 0x000002921ce8d7f0 JavaThread "Thread-48" [_thread_blocked, id=12096, stack(0x000000bc0ab00000,0x000000bc0ac00000)] 0x000002921ce8e410 JavaThread "Thread-49" [_thread_blocked, id=40720, stack(0x000000bc0ac00000,0x000000bc0ad00000)] 0x000002921ce8a160 JavaThread "Thread-50" [_thread_blocked, id=13512, stack(0x000000bc0ad00000,0x000000bc0ae00000)] 0x000002921ce91490 JavaThread "Thread-51" [_thread_blocked, id=50144, stack(0x000000bc0ae00000,0x000000bc0af00000)] 0x000002921ce8f640 JavaThread "Thread-52" [_thread_blocked, id=42784, stack(0x000000bc0af00000,0x000000bc0b000000)] 0x000002921ce90260 JavaThread "Thread-53" [_thread_blocked, id=19292, stack(0x000000bc0b000000,0x000000bc0b100000)] 0x000002921ce8b390 JavaThread "Thread-54" [_thread_blocked, id=14160, stack(0x000000bc0b100000,0x000000bc0b200000)] 0x000002921ce8b9a0 JavaThread "Thread-55" [_thread_blocked, id=38204, stack(0x000000bc0b200000,0x000000bc0b300000)] 0x000002921ce90870 JavaThread "Thread-56" [_thread_blocked, id=27736, stack(0x000000bc0b300000,0x000000bc0b400000)] 0x000002921ce91aa0 JavaThread "Thread-57" [_thread_blocked, id=10424, stack(0x000000bc0b400000,0x000000bc0b500000)] 0x000002921ce8a770 JavaThread "Thread-58" [_thread_blocked, id=2184, stack(0x000000bc0b500000,0x000000bc0b600000)] 0x000002921ce90e80 JavaThread "Thread-59" [_thread_blocked, id=22508, stack(0x000000bc0b600000,0x000000bc0b700000)] Other Threads: =>0x000002927aa6e960 VMThread "VM Thread" [stack: 0x000000bc06200000,0x000000bc06300000] [id=23788] 0x000002927bb7fef0 WatcherThread "VM Periodic Task Thread" [stack: 0x000000bc06c00000,0x000000bc06d00000] [id=8916] 0x0000029278471890 WorkerThread "GC Thread#0" [stack: 0x000000bc06100000,0x000000bc06200000] [id=27700] 0x000002921ceb0b60 WorkerThread "GC Thread#1" [stack: 0x000000bc09600000,0x000000bc09700000] [id=48256] 0x000002921cea2b10 WorkerThread "GC Thread#2" [stack: 0x000000bc09700000,0x000000bc09800000] [id=9788] 0x000002921c735980 WorkerThread "GC Thread#3" [stack: 0x000000bc09800000,0x000000bc09900000] [id=9468] 0x000002921cbdf060 WorkerThread "GC Thread#4" [stack: 0x000000bc09900000,0x000000bc09a00000] [id=52536] 0x000002921cbdf380 WorkerThread "GC Thread#5" [stack: 0x000000bc09a00000,0x000000bc09b00000] [id=56164] 0x000002921c105b80 WorkerThread "GC Thread#6" [stack: 0x000000bc09b00000,0x000000bc09c00000] [id=55336] 0x000002921c1062b0 WorkerThread "GC Thread#7" [stack: 0x000000bc09c00000,0x000000bc09d00000] [id=44164] 0x000002921c1069d0 WorkerThread "GC Thread#8" [stack: 0x000000bc09d00000,0x000000bc09e00000] [id=28296] 0x000002921c106cf0 WorkerThread "GC Thread#9" [stack: 0x000000bc09e00000,0x000000bc09f00000] [id=11832] Threads with active compile tasks: C2 CompilerThread0 14433 1574 4 jdk.internal.math.DoubleToDecimal::append8Digits (42 bytes) C2 CompilerThread1 14433 2247 4 java.lang.invoke.LambdaForm$MH/0x00000008011da800::invoke (86 bytes) C2 CompilerThread2 14433 1650 4 java.lang.Integer::formatUnsignedInt (40 bytes) VM state: at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event]) [0x0000029278446a70] Threads_lock - owner thread: 0x000002927aa6e960 Heap address: 0x00000000c0000000, size: 1024 MB, Compressed Oops mode: 32-bit CDS archive(s) mapped at: [0x0000000800000000-0x0000000800ce0000-0x0000000800ce0000), size 13500416, SharedBaseAddress: 0x0000000800000000, ArchiveRelocationMode: 0. Compressed class space mapped at: 0x0000000801000000-0x0000000841000000, reserved size: 1073741824 Narrow klass base: 0x0000000800000000, Narrow klass shift: 0, Narrow klass range: 0x100000000 . . . . . . . . . . . . . . . .
19-10-2022
One more hs_err log on Windows: # # Internal Error (c:\sb\prod\1665797520\workspace\open\src\hotspot\share\oops/compressedOops.inline.hpp:135), pid=31072, tid=5952 # assert(check_alignment(result)) failed: address not aligned: 0x0000000800000001 # # JRE version: Java(TM) SE Runtime Environment (20.0) (fastdebug build 20-internal-2022-10-15-0130374.serguei.spitsyn.jdk20.1) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-internal-2022-10-15-0130374.serguei.spitsyn.jdk20.1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, windows-amd64) # Core dump will be written. Default location: C:\sb\prod\1666164744\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\scratch\0\hs_err_pid31072.mdmp # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # --------------- S U M M A R Y ------------ Command Line: -XX:MaxRAMPercentage=4.16667 -Djava.io.tmpdir=c:\sb\prod\1666164744\testoutput\test-support\jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_RedefineClasses_StressRedefine\tmp -XX:+CreateCoredumpOnCrash -XX:+UseParallelGC -XX:+UseNUMA -Djava.library.path=c:\ade\mesos\work_dir\jib-master\install\2022-10-15-0130374.serguei.spitsyn.jdk20.1\windows-x64-debug.test\hotspot\jtreg\native -agentlib:stressRedefine nsk.jvmti.RedefineClasses.StressRedefine ./bin Host: win2016-x64-280751, AMD EPYC 7J13 64-Core Processor , 12 cores, 23G, Windows Server 2016 , 64 bit Build 14393 (10.0.14393.3630) Time: Wed Oct 19 08:34:38 2022 /GM elapsed time: 11.616775 seconds (0d 0h 0m 11s) --------------- T H R E A D --------------- Current thread (0x000002756738e220): VMThread "VM Thread" [stack: 0x000000f06d900000,0x000000f06da00000] [id=5952] Stack: [0x000000f06d900000,0x000000f06da00000] Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xc3ad41] os::win32::platform_print_native_stack+0xf1 (os_windows_x86.cpp:236) V [jvm.dll+0xe8db97] VMError::report+0x10e7 (vmError.cpp:838) V [jvm.dll+0xe8f72e] VMError::report_and_die+0x65e (vmError.cpp:1686) V [jvm.dll+0xe8fea4] VMError::report_and_die+0x64 (vmError.cpp:1467) V [jvm.dll+0x583027] report_vm_error+0xb7 (debug.cpp:284) V [jvm.dll+0x18eaf] oopDesc::klass+0x9f (oop.inline.hpp:86) V [jvm.dll+0x7eeaf6] java_lang_Class::set_klass+0x76 (javaClasses.cpp:1472) V [jvm.dll+0x79eeff] InstanceKlass::deallocate_contents+0x6f (instanceKlass.cpp:585) V [jvm.dll+0x4e29d1] ClassLoaderData::free_deallocate_list+0x381 (classLoaderData.cpp:854) V [jvm.dll+0x4e5e7d] ClassLoaderDataGraph::clean_deallocate_lists+0x7d (classLoaderDataGraph.cpp:164) V [jvm.dll+0xa0e165] VM_RedefineClasses::doit+0x1f5 (jvmtiRedefineClasses.cpp:304) V [jvm.dll+0xe96f2d] VM_Operation::evaluate+0xcd (vmOperations.cpp:72) V [jvm.dll+0xe98b26] VMThread::evaluate_operation+0xa6 (vmThread.cpp:283) V [jvm.dll+0xe992c6] VMThread::inner_execute+0x256 (vmThread.cpp:432) V [jvm.dll+0xe99630] VMThread::run+0x150 (vmThread.cpp:175) V [jvm.dll+0xe079b7] Thread::call_run+0x257 (thread.cpp:229) V [jvm.dll+0xc39688] thread_native_entry+0xb8 (os_windows.cpp:547) C [ucrtbase.dll+0x1fb80] C [KERNEL32.DLL+0x84d4] C [ntdll.dll+0x51791] VM_Operation (0x000000f070cfef00): RedefineClasses, mode: safepoint, requested by thread 0x000002756f427ec0, redefining class MyClass
19-10-2022
It looks like this test is not healthy and allows JVMTI RedefineClasses to silently fail. We can see in the test log that he RedefineClasses returned many error codes which are not checked at all: JVMTI_ERROR_FAILS_VERIFICATION JVMTI_ERROR_INVALID_CLASS_FORMAT JVMTI_ERROR_UNSUPPORTED_REDEFINITION_METHOD_DELETED I'm not sure yet why is it so. It happens in the native method makeRedefinition(): err = jvmti->RedefineClasses(1, &classDef); if (err != JVMTI_ERROR_NONE) { printf("%s: Failed to call RedefineClasses():\n", __FILE__); printf("\tthe function returned error %d: %s\n", err, TranslateError(err)); printf("\tFor more info about this error see the JVMTI spec.\n"); return STATUS_FAILED; <== !!! STATUS_FAILED is returned here !!! } However, the result value from this method is never checked: public void run() { while (stresser.continueExecution()) { byte[] badBytecode = bytecode.clone(); if (random.nextDouble() < corruptingBytecodeProbability) { badBytecode[random.nextInt(bytecode.length)] = 42; } makeRedefinition(2, myClass, badBytecode); <== ??? Failing status not checked ??? }
19-10-2022