JDK-8276177 : nsk/jvmti/RedefineClasses/StressRedefineWithoutBytecodeCorruption failed with "assert(def_ik->is_being_redefined()) failed: should be being redefined to get here"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jvmti
  • Affected Version: 11,17,18
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: aarch64
  • Submitted: 2021-10-29
  • Updated: 2021-12-22
  • Resolved: 2021-11-17
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 17 JDK 18
11.0.15-oracleFixed 17.0.3-oracleFixed 18 b25Fixed
Related Reports
Duplicate :  
JDK-8272800 - vmTestbase/nsk/jvmti/RedefineClasses/StressRedefineWithoutBytecodeCorruption/TestDescription.java timing out
Relates :  
JDK-8276367 - ProblemList vmTestbase/nsk/jvmti/RedefineClasses/StressRedefineWithoutBytecodeCorruption/TestDescription.java
Relates :  
JDK-8277216 - Examine InstanceKlass::_misc_flags for concurrency issues
Relates :  
JDK-8055008 - Clean up code that saves the previous versions of redefined classes
Description
The following test failed in the JDK18 CI:

vmTestbase/nsk/jvmti/RedefineClasses/StressRedefineWithoutBytecodeCorruption/TestDescription.java

Here's a snippet from the log file:

[273.575s][trace][redefine,class,iklass,purge] previous version stats: live=32, deleted=1
[273.575s][trace][redefine,class,iklass,add  ] scratch class added; one of its methods is on_stack.
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/jvmtiRedefineClasses.cpp:155
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/opt/mach5/mesos/work_dir/slaves/35546316-ed70-4084-8244-d672e7734fd1-S13253/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/a29dbdc8-d4b1-441d-bd87-8509bf4f3b99/runs/335c866a-b223-4ddd-a8da-d156bdb71db5/workspace/open/src/hotspot/share/prims/jvmtiRedefineClasses.cpp:155), pid=2742125, tid=2742507
#  assert(def_ik->is_being_redefined()) failed: should be being redefined to get here
#
# JRE version: Java(TM) SE Runtime Environment (18.0+22) (fastdebug build 18-ea+22-1368)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 18-ea+22-1368, compiled mode, sharing, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# V  [libjvm.so+0x118073c]  VM_RedefineClasses::unlock_classes()+0x2cc
#
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/a2dc162d-743b-4800-9e92-31f85abb45b1-S33720/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/25065e51-582f-41ab-9753-e9c384c77bf7/runs/557e56f4-ee33-477c-8b83-d62f63b6b6b5/testoutput/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_quick/scratch/0/core.2742125)
#
# An error report file with more information is saved as:
# /opt/mach5/mesos/work_dir/slaves/a2dc162d-743b-4800-9e92-31f85abb45b1-S33720/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/25065e51-582f-41ab-9753-e9c384c77bf7/runs/557e56f4-ee33-477c-8b83-d62f63b6b6b5/testoutput/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_quick/scratch/0/hs_err_pid2742125.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
<<<<<<<< RedefineClasses() is successfully done
>>>>>>>> Invoke RedefineClasses():
	new class byte count=2286
----------System.err:(0/0)----------
----------rerun:(45/8571)*----------

Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x0000fffd60133a90):  JavaThread "Thread-22" [_thread_in_vm, id=2742507, stack(0x0000fffd03c00000,0x0000fffd03e00000)]

Stack: [0x0000fffd03c00000,0x0000fffd03e00000],  sp=0x0000fffd03dfdfa0,  free space=2039k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x118073c]  VM_RedefineClasses::unlock_classes()+0x2cc
V  [libjvm.so+0x1182564]  VM_RedefineClasses::doit_epilogue()+0x14
V  [libjvm.so+0x186bcc8]  VMThread::execute(VM_Operation*)+0xa8
V  [libjvm.so+0x11450cc]  JvmtiEnv::RedefineClasses(int, jvmtiClassDefinition const*)+0x6c
V  [libjvm.so+0x10e4054]  jvmti_RedefineClasses+0x114
C  [libstressRedefine.so+0xbf70]  Java_nsk_jvmti_RedefineClasses_StressRedefine_makeRedefinition+0xe0
J 9328  nsk.jvmti.RedefineClasses.StressRedefine.makeRedefinition(ILjava/lang/Class;[B)I (0 bytes) @ 0x0000fffdbd141f64 [0x0000fffdbd141ec0+0x00000000000000a4]
C  0x0000fffd6b646af7

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 9328  nsk.jvmti.RedefineClasses.StressRedefine.makeRedefinition(ILjava/lang/Class;[B)I (0 bytes) @ 0x0000fffdbd141f64 [0x0000fffdbd141ec0+0x00000000000000a4]
j  nsk.jvmti.RedefineClasses.StressRedefine$Worker.run()V+55
J 1893 c2 java.lang.Thread.run()V java.base@18-ea (17 bytes) @ 0x0000fffdbc46abb0 [0x0000fffdbc46ab40+0x0000000000000070]
v  ~StubRoutines::call_stub


This test also has another unresolved bug:

JDK-8272800 vmTestbase/nsk/jvmti/RedefineClasses/StressRedefineWithoutBytecodeCorruption/TestDescription.java timing out
Comments
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk17u-dev/pull/15 Date: 2021-12-20 16:05:30 +0000
20-12-2021
Fix Request (11u) A clean backport for parity with Oracle 11.0.15. StressRedefineWithoutBytecodeCorruption.java test now passed with the patch.
10-12-2021
Fix Request (17u) A clean backport for parity with Oracle 17.0.3. StressRedefineWithoutBytecodeCorruption.java test now passed with the patch.
06-12-2021
Changeset: a907b2b1 Author: Coleen Phillimore <coleenp@openjdk.org> Date: 2021-11-17 19:53:55 +0000 URL: https://git.openjdk.java.net/jdk/commit/a907b2b144f2af27392eb7c2f9656fbb1a759618
17-11-2021
Yes, I remembered the _is_marked_dependent issue which made me suspicious of this flag. Unfortunately adding another bool for is_being_redefined will leave an alignment gap in InstanceKlass, which is why I chose the atomic operations in AccessFlags to fix this. The other misc flags should be examined and fixed, whatever the fix is going to be, under a different RFE in order to make this small backport easier.
16-11-2021
Code introduced in JDK 9 under JDK-8055008, but also backported to 8u.
16-11-2021
I would say _misc_is_being_redefined should not be part of the _misc flags, the same way that we have: // _is_marked_dependent can be set concurrently, thus cannot be part of the // _misc_flags. bool _is_marked_dependent; all misc flags should be set once at "construction" time. We've been bitten by bitfields before.
15-11-2021
So what's happening is one thread is setting the flag in InstanceKlass: void set_has_resolved_methods() { _misc_flags |= _misc_has_resolved_methods; } Through this path: resolve_MemberName init_method_MemberName CallInfo::CallInfo(method) set_resolved_method_name() java_lang_invoke_ResolvedMethodName::find_resolved_method() -> InstanceKlass::set_has_resolved_methods() While the other threads are setting and resetting the _misc_is_being_redefined bit. Since the & isn't thread safe, the _misc_is_being_redefined bit is getting lost (or reset in the case of the linked bugs). In the core file: (gdb) print /x def_ik->_misc_flags $4 = 0x2087 Which 0x2000 is the _misc_has_resolved_methods flag. I checked the other _misc flags and most are only set during classfile parsing. So other than _misc_is_being_redefined, the bits are write once. We should file an RFE so the setter functions don't look like the values can be toggled, and make sure no other misc flag can reset the value of another concurrently. It's possible that reflection using MethodHandles may have made this old bug more frequent, but I can't see that from the core file.
15-11-2021
You only need explicit orderAccess operation if the flag is being accessed lock-free. Otherwise the locking provides acquire-release semantics.
02-11-2021
There must be a memory ordering problem with aarch64. There are lots of threads waiting in lock_classes if (ik->is_being_redefined()) { ml.wait(); has_redefined = true; break; // for loop and the one thread that crashes has: InstanceKlass* def_ik = get_ik(_class_defs[i].klass); if (redef_classes->length() > 0) { // Remove the class from _classes_being_redefined list Klass* k = redef_classes->pop(); assert(def_ik == k, "unlocking wrong class"); } assert(def_ik->is_being_redefined(), "should be being redefined to get here"); // Unlock after we finish all redefines for this class within // the thread. Same class can be pushed to the list multiple // times (not more than once by each recursive redefinition). if (!redef_classes->contains(def_ik)) { def_ik->set_is_being_redefined(false); } def_ik and ik above are the same class. The is_being_redefined flag is false. Both lock_classes and unlock_classes are holding or waiting on the RedefineClasses_lock, so there's no order access load/stores around the is_being_redefined flag.
02-11-2021
ILW = HLM = P3
02-11-2021
This test is being ProblemListed for the hanging failure mode using the following bug: JDK-8276367 ProblemList vmTestbase/nsk/jvmti/RedefineClasses/StressRedefineWithoutBytecodeCorruption/TestDescription.java That ProblemListing will mean that investigating this bug will require UnProblemListing this test.
02-11-2021
The jdk-18+22-1388-tier4 sighting has a different failure mode. Here's a snippet from the log file: [379.955s][trace][redefine,class,iklass,purge] previous version 0x0000000801149228 is alive [379.955s][trace][redefine,class,iklass,purge] previous version stats: live=192, deleted=0 [379.955s][trace][redefine,class,iklass,add ] scratch class not added; no methods are running # To suppress the following error report, specify this argument # after -XX: or in .hotspotrc: SuppressErrorAt=/jvmtiRedefineClasses.cpp:155 # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (/opt/mach5/mesos/work_dir/slaves/35546316-ed70-4084-8244-d672e7734fd1-S13245/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/743ef649-3cf8-4c60-9469-03662d3c6ad2/runs/8776132f-06f6-47f2-807e-8ee3c630e6d9/workspace/open/src/hotspot/share/prims/jvmtiRedefineClasses.cpp:155), pid=2722366, tid=2722718 # assert(def_ik->is_being_redefined()) failed: should be being redefined to get here # # JRE version: Java(TM) SE Runtime Environment (18.0+22) (fastdebug build 18-ea+22-1388) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 18-ea+22-1388, compiled mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64) # Problematic frame: # V [libjvm.so+0x118065c] VM_RedefineClasses::unlock_classes()+0x2cc # # Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/35546316-ed70-4084-8244-d672e7734fd1-S13289/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/3e417405-6842-44ca-a558-0d2e76357045/runs/bf3b8517-685b-4f26-9efe-7a4d0f4f95bf/testoutput/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_quick/scratch/3/core.2722366) # # An error report file with more information is saved as: # /opt/mach5/mesos/work_dir/slaves/35546316-ed70-4084-8244-d672e7734fd1-S13289/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/3e417405-6842-44ca-a558-0d2e76357045/runs/bf3b8517-685b-4f26-9efe-7a4d0f4f95bf/testoutput/test-support/jtreg_open_test_hotspot_jtreg_vmTestbase_nsk_jvmti_quick/scratch/3/hs_err_pid2722366.log # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # <<<<<<<< RedefineClasses() is successfully done >>>>>>>> Invoke RedefineClasses(): new class byte count=2286 ----------System.err:(0/0)---------- ----------rerun:(45/8572)*---------- Here's the crashing thread's stack: --------------- T H R E A D --------------- Current thread (0x0000fffbe841d8b0): JavaThread "Thread-34" [_thread_in_vm, id=2722718, stack(0x0000fffbac920000,0x0000fffbacb20000)] Stack: [0x0000fffbac920000,0x0000fffbacb20000], sp=0x0000fffbacb1dfa0, free space=2039k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x118065c] VM_RedefineClasses::unlock_classes()+0x2cc V [libjvm.so+0x1182484] VM_RedefineClasses::doit_epilogue()+0x14 V [libjvm.so+0x186c098] VMThread::execute(VM_Operation*)+0xa8 V [libjvm.so+0x1144fec] JvmtiEnv::RedefineClasses(int, jvmtiClassDefinition const*)+0x6c V [libjvm.so+0x10e3f74] jvmti_RedefineClasses+0x114 C [libstressRedefine.so+0xbf70] Java_nsk_jvmti_RedefineClasses_StressRedefine_makeRedefinition+0xe0 J 17886 nsk.jvmti.RedefineClasses.StressRedefine.makeRedefinition(ILjava/lang/Class;[B)I (0 bytes) @ 0x0000fffc4e39dd64 [0x0000fffc4e39dcc0+0x00000000000000a4] C 0x0000fffc20fe27f7 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 17886 nsk.jvmti.RedefineClasses.StressRedefine.makeRedefinition(ILjava/lang/Class;[B)I (0 bytes) @ 0x0000fffc4e39dd64 [0x0000fffc4e39dcc0+0x00000000000000a4] j nsk.jvmti.RedefineClasses.StressRedefine$Worker.run()V+55 J 3575 c2 java.lang.Thread.run()V java.base@18-ea (17 bytes) @ 0x0000fffc4d49c430 [0x0000fffc4d49c3c0+0x0000000000000070] v ~StubRoutines::call_stub
01-11-2021