JDK-8155951 : VM crash in nsk/jvmti/RedefineClasses/StressRedefine: assert failed: Corrupted constant pool
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jvmti
  • Affected Version: 8,9
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2016-05-03
  • Updated: 2019-10-04
  • Resolved: 2016-05-17
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 8 JDK 9 Other
8u212Fixed 9 b120Fixed openjdk8u232Fixed
Related Reports
Relates :  
JDK-6227506 - JVMTI Spec: Atomicity of RedefineClasses should be specified
Relates :  
JDK-8151066 - assert(0 <= i && i < length()) failed: index out of bounds
Relates :  
JDK-5088035 - hotswap fires assert(0 <= i && i < length(),"index out of bounds")
Description
When running StressRedefine with TraceRedefineClasses on we can hit this assert.
It's seem like much bigger chans of getting this error if you run more test and concurrent.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/home/rehn/source/jdk/vanilla-hs/hotspot/src/share/vm/oops/constantPool.hpp:498), pid=23533, tid=23761
#  assert(tag_at(which).is_invoke_dynamic()) failed: Corrupted constant pool
#
# JRE version: Java(TM) SE Runtime Environment (9.0) (fastdebug build 9-internal+0-2016-05-03-153448.rehn.vanilla-hs)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 9-internal+0-2016-05-03-153448.rehn.vanilla-hs, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Core dump will be written. Default location: /var/cores/core.%e.%p.%h.%t
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -XX:TraceRedefineClasses=2147475455 -Xlog:disable -Xlog:all=warning:file=warnerr.%p.log::filecount=0 -agentlib:stressRedefine nsk.jvmti.RedefineClasses.StressRedefine /home/rehn/source/jdk/vanilla-hs/build/linux-x86_64-normal-server-fastdebug/images/test/hotspot/closed/tonga/bin

Host: rehn-ws, Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz, 32 cores, 31G, Fedora release 23 (Twenty Three)
Time: Tue May  3 16:02:49 2016 CEST elapsed time: 32 seconds (0d 0h 0m 32s)

---------------  T H R E A D  ---------------

Current thread (0x00007fa7a03bb800):  JavaThread "C2 CompilerThread0" daemon [_thread_in_vm, id=23761, stack(0x00007fa69b3fc000,0x00007fa69b4fd000)]


Current CompileTask:
C2:  32220 6923   !   4       jdk.internal.reflect.GeneratedMethodAccessor355::invoke (286 bytes)

Stack: [0x00007fa69b3fc000,0x00007fa69b4fd000],  sp=0x00007fa69b4f9290,  free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x146fa9e]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x47e
V  [libjvm.so+0x14704ff]  VMError::report_and_die(Thread*, char const*, int, char const*, char const*, __va_list_tag*)+0x2f
V  [libjvm.so+0x950df0]  report_vm_error(char const*, int, char const*, char const*, ...)+0xe0
V  [libjvm.so+0x941107]  ConstantPool::invoke_dynamic_name_and_type_ref_index_at(int)+0x37
V  [libjvm.so+0x937e78]  ConstantPool::impl_name_and_type_ref_index_at(int, bool)+0x138
V  [libjvm.so+0x7e8308]  ciBytecodeStream::get_method_signature_index()+0x498
V  [libjvm.so+0x7e8a30]  ciBytecodeStream::get_method(bool&, ciSignature**)+0x550
V  [libjvm.so+0x7a7a43]  ciMethod::get_method_at_bci(int, bool&, ciSignature**)+0x143
V  [libjvm.so+0xa3ece9]  Parse::do_call()+0x509
V  [libjvm.so+0x11adae9]  Parse::do_one_bytecode()+0x4f9
V  [libjvm.so+0x11a1a4d]  Parse::do_one_block()+0x40d
V  [libjvm.so+0x11a214c]  Parse::do_all_blocks()+0x21c
V  [libjvm.so+0x11a3c75]  Parse::Parse(JVMState*, ciMethod*, float)+0xba5
V  [libjvm.so+0x7300c8]  ParseGenerator::generate(JVMState*)+0x138
V  [libjvm.so+0xa3ebc1]  Parse::do_call()+0x3e1
V  [libjvm.so+0x11adae9]  Parse::do_one_bytecode()+0x4f9
V  [libjvm.so+0x11a1a4d]  Parse::do_one_block()+0x40d
V  [libjvm.so+0x11a214c]  Parse::do_all_blocks()+0x21c
V  [libjvm.so+0x11a3c75]  Parse::Parse(JVMState*, ciMethod*, float)+0xba5
V  [libjvm.so+0x7300c8]  ParseGenerator::generate(JVMState*)+0x138
V  [libjvm.so+0x8d8369]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, DirectiveSet*)+0xcd9
V  [libjvm.so+0x72ea0b]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0x2bb
V  [libjvm.so+0x8e2661]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x491
V  [libjvm.so+0x8e331b]  CompileBroker::compiler_thread_loop()+0x37b
V  [libjvm.so+0x13df66e]  JavaThread::thread_main_inner()+0x15e
V  [libjvm.so+0x13df84d]  JavaThread::run()+0x16d
V  [libjvm.so+0x1151b12]  java_start(Thread*)+0x112
C  [libpthread.so.0+0x760a]  start_thread+0xca
Comments
Fix Request (8u) I would like to backport this patch to 8u, as it is on Oracle 8u backport list. The patch does not apply cleanly. 8u code review thread: https://mail.openjdk.java.net/pipermail/jdk8u-dev/2019-August/010024.html (Reviewed)
20-08-2019
[to Coleen: ] Great! It seems, you identified the root cause of this issue.
11-05-2016
So JDK-8151066 is due to not thread safe constant pool merging. This is due to the merged constant pools do not produce compatible constant pool caches, because invokedynamic call site entries are appended to the end of the constant pool. The old method points to a cpCache entry for the old constant pool cache. The lookup uses the merged constant pool cpCache: int ciBytecodeStream::get_method_signature_index() { GUARDED_VM_ENTRY( ConstantPool* cpool = _holder->get_instanceKlass()->constants(); const int method_index = get_method_index(); <= cp cache index for old constant pool const int name_and_type_index = cpool->name_and_type_ref_index_at(method_index, _method->get_Method()->constants()); >>> calls: >>> int pool_index = invokedynamic_cp_cache_entry_at(which)->constant_pool_index(); >>> but (!tag_at(pool_index).is_invoke_dynamic()) >>> return cpool->signature_ref_index_at(name_and_type_index); ) } We don't rewrite the old method bytecodes because we assume that the cpCache indices compare but for invokedynamic they don't. With the INDY string concatenation, there is more constant pool merging and more invokedynamic cpCache entries created.
11-05-2016
Serguei, I like this idea! This way we won't hold the lock during safepoint so we can use a vm Monitor.
10-05-2016
[to David H.: ] If you are asking about the suggestion #5 then the RedefineClasses_lock must be unlocked before safepoint. It has to be locked in doit_prologue() and doit_epilogue() to change the _misc_class_is_being_redefined bits of the target classes.
10-05-2016
I have a suggestion #5: 5. Add one more bit flag into the InstanceKlass::_misc_flags: _misc_class_is_being_redefined = 1 << 14, // class is in process of redefinition In the VM_RedefineClasses::doit_prologue() with protection of new monitor RedefineClasses_lock check if all classes involved into redefinition do not have the bit _misc_class_is_being_redefined. Loop in waiting on the monitor if any of the target classes are in process of redefinition. Otherwise, set the _misc_class_is_being_redefined for all target classes and release the monitor. At the VM_RedefineClasses::doit_epilogue() with protection of the monitor RedefineClasses_lock release the bit _misc_class_is_being_redefined for all classes involved into redefinition and notify all the monitor waiters. There can be some variations of this approach related to the wait/notify protocol.
10-05-2016
More a comment on #1 and #2 - thanks for clarifying #5.
10-05-2016
I'm unclear how a race originates if we are at a safepoint? Anything using locking while at a safepoint is fragile at best.
10-05-2016
#1 won't work. Can't merge constant pools in the vm thread. #4 seems to work, still testing.
09-05-2016
4. detect that race occurred in VM_RedefineClasses::doit() with now corrected constant pool version number and give JVMTI_ERROR_INTERNAL if the racing constant pools don't match. I've tested a version of this for the last bug but decided to be more conservative for that fix.
09-05-2016
I'm convinced the merging for constant pools works properly for invokedynamic. The bytecode corruption causes the operands to get mangled so they don't match and require a new constant pool entry for invokedynamic. Some enhanced printing at the failure: - 69 : MethodHandle : ref_kind=6 - 6 : String : 'asdfsdf-220165115sdf' ref_index=92 - 92 : Method : klass_index=109 name_and_type_index=110 - 69 : MethodHandle : ref_kind=6 - 6 : String : 'asdfsdf1276958756sdf' ref_index=92 - 92 : Method : klass_index=109 name_and_type_index=110 Since there are so many things to match for JVM_CONSTANT_InvokeDynamic constant pool type, there is likely some difference causing a new constant pool entry to be added to the merged constant pool. So this problem is more likely with INDY string concatenation, which is why we're seeing it more. There are many threads merging constant pools at once, especially on this machine I'm running it on.
06-05-2016
From inside ciMethod.cpp call: (gdb) print *((Method*)_method->_metadata)->_constMethod->_constants $18 = {<Metadata> = {<MetaspaceObj> = {<No data fields>}, _vptr.Metadata = 0x7eff3a72a750 <vtable for ConstantPool+16>, _valid = 0}, _tags = 0x7efbd28dfe40, _cache = 0x7efbd2b30040, _pool_holder = 0x800191c30, _operands = 0x7efbd075ad10, _resolved_references = 0x7efc08009070, _reference_map = 0x7efbd28e1178, _flags = 0, _length = 4913, _saved = { _resolved_reference_length = 341, _version = 341}} then down we get constant pool from holder 468 int ciBytecodeStream::get_method_signature_index() { 469 GUARDED_VM_ENTRY( 470 ConstantPool* cpool = _holder->get_instanceKlass()->constants(); Which is: (gdb) print *this $19 = {<Metadata> = {<MetaspaceObj> = {<No data fields>}, _vptr.Metadata = 0x7eff3a72a750 <vtable for ConstantPool+16>, _valid = 0}, _tags = 0x7efbd1853e60, _cache = 0x7efbd33937a8, _pool_holder = 0x800191c30, _operands = 0x7efbd2e90c28, _resolved_references = 0x7efc00001100, _reference_map = 0x7efb9ff156d8, _flags = 0, _length = 407, _saved = { _resolved_reference_length = 21, _version = 21}} Likely parallel merging of constant pools and the version 21 won, but the length of the one pointed to by the old method is suspicious 4913. I think with INDY string concatenation, there are more invokedynamic constant pool indices that don't merge and require new entries. So there are likely two problems here: parallel merging constant pools and inability to match invokedynamic entries (which may not be a problem, since there are several sub-indices that have to match).
04-05-2016
I can reproduce this on a bigger system than I had.
04-05-2016
I reproduced with vanilla jdk9/hs (hotspot hg head 11109:fc5b64f70199) and a jib build. (I have not been able to reproduce in aurora, but it is without TEST_CONCURRENCY which I set on local tonga runs)
03-05-2016
The hs_err_pid_15670 is not a vanilla stree, it contains changes from 8153535, but from a jprt build. The other one is vanilla, but local build without jib, with hg head: changeset: 11109:fc5b64f70199
03-05-2016