JDK-8210457 : JVM crash in ResolvedMethodTable::add_method(Handle)
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 11
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2018-09-03
  • Updated: 2019-08-15
  • Resolved: 2019-02-26
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 12 JDK 13
11.0.4Fixed 12.0.2Fixed 13 b10Fixed
Related Reports
Duplicate :  
Relates :  
Relates :  
Description

# Problematic frame:
# V  [libjvm.so+0xcf2a79]  ResolvedMethodTable::add_method(Handle)+0x49

Stack: [0x00007f6e755ed000,0x00007f6e757ee000],  sp=0x00007f6e757ea2a0,  free space=2036k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xcf2a79]  ResolvedMethodTable::add_method(Handle)+0x49
V  [libjvm.so+0x8bad3d]  java_lang_invoke_ResolvedMethodName::find_resolved_method(methodHandle const&, Thread*)+0xcd
V  [libjvm.so+0xa90e1b]  CallInfo::CallInfo(Method*, Klass*, Thread*)+0x2db
V  [libjvm.so+0x8b8d31]  java_lang_StackFrameInfo::set_method_and_bci(Handle, methodHandle const&, int, Thread*)+0x71
V  [libjvm.so+0xd4b1cf]  JavaFrameStream::fill_frame(int, objArrayHandle, methodHandle const&, Thread*)+0x10f
V  [libjvm.so+0xd4c5c0]  StackWalk::fill_in_frames(long, BaseFrameStream&, int, int, objArrayHandle, int&, Thread*)+0xe0
V  [libjvm.so+0xd4cc0d]  StackWalk::fetchNextBatch(Handle, long, long, int, int, objArrayHandle, Thread*)+0x16d
V  [libjvm.so+0x95fc75]  JVM_MoreStackWalk+0xc5
J 5913  java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(JJII[Ljava/lang/Object;)I java.base@11-ea (0 bytes) @ 0x00007f7284108b20 [0x00007f7284108a40+0x00000000000000e0]
J 25016 c2 java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(I)I java.base@11-ea (121 bytes) @ 0x00007f72855b21e4 [0x00007f72855b2160+0x0000000000000084]
J 23829 c2 java.lang.StackStreamFactory$AbstractStackWalker.peekFrame()Ljava/lang/Class; java.base@11-ea (132 bytes) @ 0x00007f728594e9b4 [0x00007f728594e4e0+0x00000000000004d4]
J 15295 c2 java.lang.StackStreamFactory$StackFrameTraverser.tryAdvance(Ljava/util/function/Consumer;)Z java.base@11-ea (114 bytes) @ 0x00007f7284c8a2f4 [0x00007f7284c8a220+0x00000000000000d4]
J 24352 c2 java.util.stream.ReferencePipeline.findFirst()Ljava/util/Optional; java.base@11-ea (12 bytes) @ 0x00007f7284a0f17c [0x00007f7284a0ea60+0x000000000000071c]
J 25007 c2 java.lang.StackStreamFactory$AbstractStackWalker.doStackWalk(JIIII)Ljava/lang/Object; java.base@11-ea (82 bytes) @ 0x00007f72855ce784 [0x00007f72855cdf00+0x0000000000000884]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x8b3559]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x3b9
V  [libjvm.so+0xd4dada]  StackWalk::walk(Handle, long, int, int, int, objArrayHandle, Thread*)+0x7fa
V  [libjvm.so+0x95fb02]  JVM_CallStackWalk+0xf2
J 5904  java.lang.StackStreamFactory$AbstractStackWalker.callStackWalk(JIII[Ljava/lang/Object;)Ljava/lang/Object; java.base@11-ea (0 bytes) @ 0x00007f7284107320 [0x00007f7284107240+0x00000000000000e0]
J 25008 c2 org.apache.logging.log4j.util.StackLocatorUtil.calcLocation(Ljava/lang/String;)Ljava/lang/StackTraceElement; (8 bytes) @ 0x00007f728561341c [0x00007f7285612ea0+0x000000000000057c]
J 18461 c2 org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (17 bytes) @ 0x00007f72842ee51c [0x00007f72842ee000+0x000000000000051c]
J 18868 c2 org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (30 bytes) @ 0x00007f72852813e0 [0x00007f7285281240+0x00000000000001a0]
J 28115 c2 org.apache.logging.log4j.spi.AbstractLogger.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V (36 bytes) @ 0x00007f7285d1bcd4 [0x00007f7285d1b7e0+0x00000000000004f4]
J 23364 c1 org.apache.logging.log4j.spi.AbstractLogger.warn(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V (17 bytes) @ 0x00007f727ca6686c [0x00007f727ca66500+0x000000000000036c]
j  de.icubic.mm.communication.icubic.IcubicDoubleBufferConnectionHandler.close()V+48
J 28435 c2 de.icubic.mm.communication.icubic.IcubicReadConnectionHandler.read(J)V (310 bytes) @ 0x00007f7285009810 [0x00007f7285008b60+0x0000000000000cb0]
J 22727 c2 de.icubic.mm.server.monitoring.overloads.JMX_IcubicDoubleBufferConnectionHandler.read(J)V (14 bytes) @ 0x00007f72854ac20c [0x00007f72854ac1c0+0x000000000000004c]
j  de.icubic.mm.communication.ConnectionManager.readWriteLoop(Ljava/nio/channels/Selector;Ljava/lang/Runnable;Ljava/lang/Object;Ljava/lang/String;)V+420
j  de.icubic.mm.communication.ConnectionManager$9.run()V+40
J 22743 c1 java.lang.Thread.run()V java.base@11-ea (17 bytes) @ 0x00007f727d5b0b2c [0x00007f727d5b09e0+0x000000000000014c]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x8b3559]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x3b9
V  [libjvm.so+0x8b16ad]  JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x1cd
V  [libjvm.so+0x95b23c]  thread_entry(JavaThread*, Thread*)+0x6c
V  [libjvm.so+0xdf961a]  JavaThread::thread_main_inner()+0x1fa
V  [libjvm.so+0xdf99a7]  JavaThread::run()+0x377
V  [libjvm.so+0xc3f010]  thread_native_entry(Thread*)+0xf0

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 5913  java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(JJII[Ljava/lang/Object;)I java.base@11-ea (0 bytes) @ 0x00007f7284108aa3 [0x00007f7284108a40+0x0000000000000063]
J 25016 c2 java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(I)I java.base@11-ea (121 bytes) @ 0x00007f72855b21e4 [0x00007f72855b2160+0x0000000000000084]
J 23829 c2 java.lang.StackStreamFactory$AbstractStackWalker.peekFrame()Ljava/lang/Class; java.base@11-ea (132 bytes) @ 0x00007f728594e9b4 [0x00007f728594e4e0+0x00000000000004d4]
J 15295 c2 java.lang.StackStreamFactory$StackFrameTraverser.tryAdvance(Ljava/util/function/Consumer;)Z java.base@11-ea (114 bytes) @ 0x00007f7284c8a2f4 [0x00007f7284c8a220+0x00000000000000d4]
J 24352 c2 java.util.stream.ReferencePipeline.findFirst()Ljava/util/Optional; java.base@11-ea (12 bytes) @ 0x00007f7284a0f17c [0x00007f7284a0ea60+0x000000000000071c]
J 25007 c2 java.lang.StackStreamFactory$AbstractStackWalker.doStackWalk(JIIII)Ljava/lang/Object; java.base@11-ea (82 bytes) @ 0x00007f72855ce784 [0x00007f72855cdf00+0x0000000000000884]
v  ~StubRoutines::call_stub
J 5904  java.lang.StackStreamFactory$AbstractStackWalker.callStackWalk(JIII[Ljava/lang/Object;)Ljava/lang/Object; java.base@11-ea (0 bytes) @ 0x00007f72841072a3 [0x00007f7284107240+0x0000000000000063]
J 25008 c2 org.apache.logging.log4j.util.StackLocatorUtil.calcLocation(Ljava/lang/String;)Ljava/lang/StackTraceElement; (8 bytes) @ 0x00007f728561341c [0x00007f7285612ea0+0x000000000000057c]
J 18461 c2 org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (17 bytes) @ 0x00007f72842ee51c [0x00007f72842ee000+0x000000000000051c]
J 18868 c2 org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (30 bytes) @ 0x00007f72852813e0 [0x00007f7285281240+0x00000000000001a0]
J 28115 c2 org.apache.logging.log4j.spi.AbstractLogger.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V (36 bytes) @ 0x00007f7285d1bcd4 [0x00007f7285d1b7e0+0x00000000000004f4]
J 23364 c1 org.apache.logging.log4j.spi.AbstractLogger.warn(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V (17 bytes) @ 0x00007f727ca6686c [0x00007f727ca66500+0x000000000000036c]
j  de.icubic.mm.communication.icubic.IcubicDoubleBufferConnectionHandler.close()V+48
J 28435 c2 de.icubic.mm.communication.icubic.IcubicReadConnectionHandler.read(J)V (310 bytes) @ 0x00007f7285009810 [0x00007f7285008b60+0x0000000000000cb0]
J 22727 c2 de.icubic.mm.server.monitoring.overloads.JMX_IcubicDoubleBufferConnectionHandler.read(J)V (14 bytes) @ 0x00007f72854ac20c [0x00007f72854ac1c0+0x000000000000004c]
j  de.icubic.mm.communication.ConnectionManager.readWriteLoop(Ljava/nio/channels/Selector;Ljava/lang/Runnable;Ljava/lang/Object;Ljava/lang/String;)V+420
j  de.icubic.mm.communication.ConnectionManager$9.run()V+40
J 22743 c1 java.lang.Thread.run()V java.base@11-ea (17 bytes) @ 0x00007f727d5b0b2c [0x00007f727d5b09e0+0x000000000000014c]
v  ~StubRoutines::call_stub



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
run our software

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
no crash
ACTUAL -
crash

FREQUENCY : occasionally
Comments
For the record, webrev links seem to be here: https://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2019-March/033035.html Coleen, are you pushing these to 12u (that would be jdk-updates/jdk12u) and 11u (that would be jdk-updates/jdk11u-dev, notice -dev)?
20-03-2019

Fix Request: Developers have been running into this crash with no workaround and have requested this to be backported. The fix is pretty simple. The patch doesn't apply cleanly because of some other cleanups in the code near it, so a trivial merge is expected with the backport. A test was developed while fixing this bug, which would be run. Other tests will be run to make sure this code didn't get broken.
12-03-2019

Returning the old deleted method could cause a crash later on. Working on a larger fix for JDK 13.
27-11-2018

The only explanation for this is that the StackWalk API is trying to create ResolvedMethodName oops for a Method* that gets deleted during redefinition, causing the method_with_idnum() to return NULL in this method. From the stack, the address crashing is 8 which is a deference of (method+_constMethod) to get the _constants to get _holder in the method_holder() call. oop ResolvedMethodTable::add_method(Handle resolved_method_name) { MutexLocker ml(ResolvedMethodTable_lock); DEBUG_ONLY(NoSafepointVerifier nsv); // Check if method has been redefined while taking out ResolvedMethodTable_lock, if so // use new method. Method* method = (Method*)java_lang_invoke_ResolvedMethodName::vmtarget(resolved_method_name()); assert(method->is_method(), "must be method"); if (method->is_old()) { // Replace method with redefined version InstanceKlass* holder = method->method_holder(); method = holder->method_with_idnum(method->method_idnum()); <== method can be NULL if deleted through redefinition java_lang_invoke_ResolvedMethodName::set_vmtarget(resolved_method_name(), method); } // Set flag in class to indicate this InstanceKlass has entries in the table // to avoid walking table during redefinition if none of the redefined classes // have any membernames in the table. method->method_holder()->set_has_resolved_methods(); return _the_table->basic_add(method, resolved_method_name); } There are redefinition events in the hs_err_pid.log file. Not being able to reproduce this, I think checking for null and returning the old method is a conservative fix.
19-11-2018

Hi, is it possible to have a core dump of this failure?
11-09-2018

Hi can you attach the hs_err_pid file and only put the top part of it in the description of the bug?
06-09-2018

Crash reported against 11 ea b23, therre is no test case or reproducible steps to verify the issue at our end. Requested submitter to provide steps or stand alone test case.
06-09-2018