Bug ID: JDK-8210457 JVM crash in ResolvedMethodTable::add

JDK-8210457 : JVM crash in ResolvedMethodTable::add_method(Handle)

Type: Bug
Component: hotspot
Sub-Component: runtime
Affected Version: 11

Priority: P3
Status: Resolved
Resolution: Fixed
OS: linux
CPU: x86_64

Submitted: 2018-09-03
Updated: 2019-08-15
Resolved: 2019-02-26

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 11	JDK 12	JDK 13
11.0.4Fixed	12.0.2Fixed	13 b10Fixed

Related Reports

Duplicate :	JDK-8223435 - JVM crash in ResolvedMethodTable::add_method
Relates :	JDK-8181171 - Deleting method for RedefineClasses breaks ResolvedMethodName
Relates :	JDK-8207024 - assert(check_klass_alignment(result)) failed: address not aligned: 0x00000008baadbabe

Description


# Problematic frame:
# V  [libjvm.so+0xcf2a79]  ResolvedMethodTable::add_method(Handle)+0x49

Stack: [0x00007f6e755ed000,0x00007f6e757ee000],  sp=0x00007f6e757ea2a0,  free space=2036k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xcf2a79]  ResolvedMethodTable::add_method(Handle)+0x49
V  [libjvm.so+0x8bad3d]  java_lang_invoke_ResolvedMethodName::find_resolved_method(methodHandle const&, Thread*)+0xcd
V  [libjvm.so+0xa90e1b]  CallInfo::CallInfo(Method*, Klass*, Thread*)+0x2db
V  [libjvm.so+0x8b8d31]  java_lang_StackFrameInfo::set_method_and_bci(Handle, methodHandle const&, int, Thread*)+0x71
V  [libjvm.so+0xd4b1cf]  JavaFrameStream::fill_frame(int, objArrayHandle, methodHandle const&, Thread*)+0x10f
V  [libjvm.so+0xd4c5c0]  StackWalk::fill_in_frames(long, BaseFrameStream&, int, int, objArrayHandle, int&, Thread*)+0xe0
V  [libjvm.so+0xd4cc0d]  StackWalk::fetchNextBatch(Handle, long, long, int, int, objArrayHandle, Thread*)+0x16d
V  [libjvm.so+0x95fc75]  JVM_MoreStackWalk+0xc5
J 5913  java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(JJII[Ljava/lang/Object;)I java.base@11-ea (0 bytes) @ 0x00007f7284108b20 [0x00007f7284108a40+0x00000000000000e0]
J 25016 c2 java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(I)I java.base@11-ea (121 bytes) @ 0x00007f72855b21e4 [0x00007f72855b2160+0x0000000000000084]
J 23829 c2 java.lang.StackStreamFactory$AbstractStackWalker.peekFrame()Ljava/lang/Class; java.base@11-ea (132 bytes) @ 0x00007f728594e9b4 [0x00007f728594e4e0+0x00000000000004d4]
J 15295 c2 java.lang.StackStreamFactory$StackFrameTraverser.tryAdvance(Ljava/util/function/Consumer;)Z java.base@11-ea (114 bytes) @ 0x00007f7284c8a2f4 [0x00007f7284c8a220+0x00000000000000d4]
J 24352 c2 java.util.stream.ReferencePipeline.findFirst()Ljava/util/Optional; java.base@11-ea (12 bytes) @ 0x00007f7284a0f17c [0x00007f7284a0ea60+0x000000000000071c]
J 25007 c2 java.lang.StackStreamFactory$AbstractStackWalker.doStackWalk(JIIII)Ljava/lang/Object; java.base@11-ea (82 bytes) @ 0x00007f72855ce784 [0x00007f72855cdf00+0x0000000000000884]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x8b3559]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x3b9
V  [libjvm.so+0xd4dada]  StackWalk::walk(Handle, long, int, int, int, objArrayHandle, Thread*)+0x7fa
V  [libjvm.so+0x95fb02]  JVM_CallStackWalk+0xf2
J 5904  java.lang.StackStreamFactory$AbstractStackWalker.callStackWalk(JIII[Ljava/lang/Object;)Ljava/lang/Object; java.base@11-ea (0 bytes) @ 0x00007f7284107320 [0x00007f7284107240+0x00000000000000e0]
J 25008 c2 org.apache.logging.log4j.util.StackLocatorUtil.calcLocation(Ljava/lang/String;)Ljava/lang/StackTraceElement; (8 bytes) @ 0x00007f728561341c [0x00007f7285612ea0+0x000000000000057c]
J 18461 c2 org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (17 bytes) @ 0x00007f72842ee51c [0x00007f72842ee000+0x000000000000051c]
J 18868 c2 org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (30 bytes) @ 0x00007f72852813e0 [0x00007f7285281240+0x00000000000001a0]
J 28115 c2 org.apache.logging.log4j.spi.AbstractLogger.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V (36 bytes) @ 0x00007f7285d1bcd4 [0x00007f7285d1b7e0+0x00000000000004f4]
J 23364 c1 org.apache.logging.log4j.spi.AbstractLogger.warn(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V (17 bytes) @ 0x00007f727ca6686c [0x00007f727ca66500+0x000000000000036c]
j  de.icubic.mm.communication.icubic.IcubicDoubleBufferConnectionHandler.close()V+48
J 28435 c2 de.icubic.mm.communication.icubic.IcubicReadConnectionHandler.read(J)V (310 bytes) @ 0x00007f7285009810 [0x00007f7285008b60+0x0000000000000cb0]
J 22727 c2 de.icubic.mm.server.monitoring.overloads.JMX_IcubicDoubleBufferConnectionHandler.read(J)V (14 bytes) @ 0x00007f72854ac20c [0x00007f72854ac1c0+0x000000000000004c]
j  de.icubic.mm.communication.ConnectionManager.readWriteLoop(Ljava/nio/channels/Selector;Ljava/lang/Runnable;Ljava/lang/Object;Ljava/lang/String;)V+420
j  de.icubic.mm.communication.ConnectionManager$9.run()V+40
J 22743 c1 java.lang.Thread.run()V java.base@11-ea (17 bytes) @ 0x00007f727d5b0b2c [0x00007f727d5b09e0+0x000000000000014c]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x8b3559]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x3b9
V  [libjvm.so+0x8b16ad]  JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x1cd
V  [libjvm.so+0x95b23c]  thread_entry(JavaThread*, Thread*)+0x6c
V  [libjvm.so+0xdf961a]  JavaThread::thread_main_inner()+0x1fa
V  [libjvm.so+0xdf99a7]  JavaThread::run()+0x377
V  [libjvm.so+0xc3f010]  thread_native_entry(Thread*)+0xf0

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 5913  java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(JJII[Ljava/lang/Object;)I java.base@11-ea (0 bytes) @ 0x00007f7284108aa3 [0x00007f7284108a40+0x0000000000000063]
J 25016 c2 java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(I)I java.base@11-ea (121 bytes) @ 0x00007f72855b21e4 [0x00007f72855b2160+0x0000000000000084]
J 23829 c2 java.lang.StackStreamFactory$AbstractStackWalker.peekFrame()Ljava/lang/Class; java.base@11-ea (132 bytes) @ 0x00007f728594e9b4 [0x00007f728594e4e0+0x00000000000004d4]
J 15295 c2 java.lang.StackStreamFactory$StackFrameTraverser.tryAdvance(Ljava/util/function/Consumer;)Z java.base@11-ea (114 bytes) @ 0x00007f7284c8a2f4 [0x00007f7284c8a220+0x00000000000000d4]
J 24352 c2 java.util.stream.ReferencePipeline.findFirst()Ljava/util/Optional; java.base@11-ea (12 bytes) @ 0x00007f7284a0f17c [0x00007f7284a0ea60+0x000000000000071c]
J 25007 c2 java.lang.StackStreamFactory$AbstractStackWalker.doStackWalk(JIIII)Ljava/lang/Object; java.base@11-ea (82 bytes) @ 0x00007f72855ce784 [0x00007f72855cdf00+0x0000000000000884]
v  ~StubRoutines::call_stub
J 5904  java.lang.StackStreamFactory$AbstractStackWalker.callStackWalk(JIII[Ljava/lang/Object;)Ljava/lang/Object; java.base@11-ea (0 bytes) @ 0x00007f72841072a3 [0x00007f7284107240+0x0000000000000063]
J 25008 c2 org.apache.logging.log4j.util.StackLocatorUtil.calcLocation(Ljava/lang/String;)Ljava/lang/StackTraceElement; (8 bytes) @ 0x00007f728561341c [0x00007f7285612ea0+0x000000000000057c]
J 18461 c2 org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (17 bytes) @ 0x00007f72842ee51c [0x00007f72842ee000+0x000000000000051c]
J 18868 c2 org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (30 bytes) @ 0x00007f72852813e0 [0x00007f7285281240+0x00000000000001a0]
J 28115 c2 org.apache.logging.log4j.spi.AbstractLogger.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V (36 bytes) @ 0x00007f7285d1bcd4 [0x00007f7285d1b7e0+0x00000000000004f4]
J 23364 c1 org.apache.logging.log4j.spi.AbstractLogger.warn(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V (17 bytes) @ 0x00007f727ca6686c [0x00007f727ca66500+0x000000000000036c]
j  de.icubic.mm.communication.icubic.IcubicDoubleBufferConnectionHandler.close()V+48
J 28435 c2 de.icubic.mm.communication.icubic.IcubicReadConnectionHandler.read(J)V (310 bytes) @ 0x00007f7285009810 [0x00007f7285008b60+0x0000000000000cb0]
J 22727 c2 de.icubic.mm.server.monitoring.overloads.JMX_IcubicDoubleBufferConnectionHandler.read(J)V (14 bytes) @ 0x00007f72854ac20c [0x00007f72854ac1c0+0x000000000000004c]
j  de.icubic.mm.communication.ConnectionManager.readWriteLoop(Ljava/nio/channels/Selector;Ljava/lang/Runnable;Ljava/lang/Object;Ljava/lang/String;)V+420
j  de.icubic.mm.communication.ConnectionManager$9.run()V+40
J 22743 c1 java.lang.Thread.run()V java.base@11-ea (17 bytes) @ 0x00007f727d5b0b2c [0x00007f727d5b09e0+0x000000000000014c]
v  ~StubRoutines::call_stub



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
run our software

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
no crash
ACTUAL -
crash

FREQUENCY : occasionally

Comments

For the record, webrev links seem to be here: https://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2019-March/033035.html Coleen, are you pushing these to 12u (that would be jdk-updates/jdk12u) and 11u (that would be jdk-updates/jdk11u-dev, notice -dev)?
20-03-2019
Fix Request: Developers have been running into this crash with no workaround and have requested this to be backported. The fix is pretty simple. The patch doesn't apply cleanly because of some other cleanups in the code near it, so a trivial merge is expected with the backport. A test was developed while fixing this bug, which would be run. Other tests will be run to make sure this code didn't get broken.
12-03-2019
Returning the old deleted method could cause a crash later on. Working on a larger fix for JDK 13.
27-11-2018
The only explanation for this is that the StackWalk API is trying to create ResolvedMethodName oops for a Method* that gets deleted during redefinition, causing the method_with_idnum() to return NULL in this method. From the stack, the address crashing is 8 which is a deference of (method+_constMethod) to get the _constants to get _holder in the method_holder() call. oop ResolvedMethodTable::add_method(Handle resolved_method_name) { MutexLocker ml(ResolvedMethodTable_lock); DEBUG_ONLY(NoSafepointVerifier nsv); // Check if method has been redefined while taking out ResolvedMethodTable_lock, if so // use new method. Method* method = (Method)java_lang_invoke_ResolvedMethodName::vmtarget(resolved_method_name()); assert(method->is_method(), "must be method"); if (method->is_old()) { // Replace method with redefined version InstanceKlass holder = method->method_holder(); method = holder->method_with_idnum(method->method_idnum()); <== method can be NULL if deleted through redefinition java_lang_invoke_ResolvedMethodName::set_vmtarget(resolved_method_name(), method); } // Set flag in class to indicate this InstanceKlass has entries in the table // to avoid walking table during redefinition if none of the redefined classes // have any membernames in the table. method->method_holder()->set_has_resolved_methods(); return _the_table->basic_add(method, resolved_method_name); } There are redefinition events in the hs_err_pid.log file. Not being able to reproduce this, I think checking for null and returning the old method is a conservative fix.
19-11-2018
Hi, is it possible to have a core dump of this failure?
11-09-2018
Hi can you attach the hs_err_pid file and only put the top part of it in the description of the bug?
06-09-2018
Crash reported against 11 ea b23, therre is no test case or reproducible steps to verify the issue at our end. Requested submitter to provide steps or stand alone test case.
06-09-2018