JDK-8016074 : NMT: assertion failed: assert(thread->thread_state() == from) failed: coming from wrong thread state
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: hs24,hs25
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: solaris
  • CPU: sparc
  • Submitted: 2013-06-06
  • Updated: 2014-01-14
  • Resolved: 2013-07-04
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7 JDK 8 Other
7u40Fixed 8Fixed hs24Fixed
Kitchensink/stability test failed due to above assertion failure, with NMT on and -XX:-AutoshutdownNMT

# A fatal error has been detected by the Java Runtime Environment:
#  Internal Error (/java/deployment/zg131198/8011968/hotspot/src/share/vm/runtime/interfaceSupport.hpp:164), pid=8838, tid=18
#  assert(thread->thread_state() == from) failed: coming from wrong thread state
# JRE version: Java(TM) SE Runtime Environment (8.0-b76) (build 1.8.0-ea-b76)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b36-internal-debug mixed mode solaris-sparc compressed oops)
# Core dump written. Default location: /tmp/zhgu/ute_run/kitchensink/core or core.8838
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp

---------------  T H R E A D  ---------------

Current thread (0x000000010045b800):  JavaThread "Reference Handler" daemon [_thread_in_Java, id=18, stack(0xffffffff3da00000,0xffffffff3db00000)]

Stack: [0xffffffff3da00000,0xffffffff3db00000],  sp=0xffffffff3dafd250,  free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1883668]  void VMError::report_and_die()+0x7f0
V  [libjvm.so+0xe5eea4]  void report_vm_error(const char*,int,const char*,const char*)+0xcc
V  [libjvm.so+0x8aa068]  void ThreadStateTransition::transition_and_fence(JavaThread*,JavaThreadState,JavaThreadState)+0x80
V  [libjvm.so+0x8a9dd4]  void ThreadStateTransition::trans_and_fence(JavaThreadState,JavaThreadState)+0x34
V  [libjvm.so+0xa5e950]  ThreadBlockInVM::ThreadBlockInVM(JavaThread*)+0x98
V  [libjvm.so+0x158bea4]  int os::sleep(Thread*,long,bool)+0x184
V  [libjvm.so+0x158c268]  void os::yield_all(int)+0x30
V  [libjvm.so+0x1477ef8]  void MemTracker::create_memory_record(unsigned char*,unsigned short,unsigned long,unsigned char*,Thread*)+0x1a0
V  [libjvm.so+0x157bf8c]  void MemTracker::record_malloc(unsigned char*,unsigned long,unsigned short,unsigned char*,Thread*)+0x114
V  [libjvm.so+0x157670c]  void*os::malloc(unsigned long,unsigned short,unsigned char*)+0x384
V  [libjvm.so+0xa09d18]  char*AllocateHeap(unsigned long,unsigned short,unsigned char*,AllocFailStrategy::AllocFailEnum)+0x98
V  [libjvm.so+0xa23274]  void*CHeapObj<(unsigned short)1792>::operator new(unsigned long,unsigned char*)+0x9c
V  [libjvm.so+0xe87d2c]  Deoptimization::UnrollBlock*Deoptimization::fetch_unroll_info_helper(JavaThread*)+0x2c
V  [libjvm.so+0xe8effc]  Deoptimization::UnrollBlock*Deoptimization::uncommon_trap(JavaThread*,int)+0x34
v  blob 0xffffffff730d7a38
J  java.lang.ref.Reference$ReferenceHandler.run()V
v  ~StubRoutines::call_stub
V  [libjvm.so+0x11620e8]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x988
V  [libjvm.so+0x158eacc]  void os::os_exception_wrapper(void(*)(JavaValue*,methodHandle*,JavaCallArguments*,Thread*),JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x44
V  [libjvm.so+0x1161740]  void JavaCalls::call(JavaValue*,methodHandle,JavaCallArguments*,Thread*)+0xd0
V  [libjvm.so+0x11607dc]  void JavaCalls::call_virtual(JavaValue*,KlassHandle,Symbol*,Symbol*,JavaCallArguments*,Thread*)+0x294
V  [libjvm.so+0x1160914]  void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*)+0x9c
V  [libjvm.so+0x1246d00]  void thread_entry(JavaThread*,Thread*)+0x108
V  [libjvm.so+0x17d1014]  void JavaThread::thread_main_inner()+0x1b4
V  [libjvm.so+0x17d0e30]  void JavaThread::run()+0x310
V  [libjvm.so+0x15820d4]  java_start+0x2cc

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  blob 0xffffffff730d7a38
J  java.lang.ref.Reference$ReferenceHandler.run()V
v  ~StubRoutines::call_stub

Apparently, os::yield_all() can only be called by JavaThread when it is in _thread_in_vm state.
SQE is OK with this backport, it's a low risk fix.

Need SQE-OK prior to approving this.

7u40-critical-request justification: Without the fix, AutoShutdownNMT=false does not work on Solaris, as the flag is one of solutions to OOM problem with Kitchensink tests. The risk is low. It redirects the yield call to system thr_yield() on Solaris, the behaviors remain the same on other platforms. The fix was reviewed by Karen Kinnear, Coleen Phillimore and Harold Seigel

Yielding NMT worker thread can be one of solution to OOM problem. Raise this priority as part of OOM fix.

Solaris yield_all() implementation has limitations, which do not exist on other platforms. The implementation depends on os::sleep(), which means that it needs JVM Thread implementation, and it blocks on Threads_lock that may cause lock out of order.

Impact: It only impact Solaris with -XX:-AutoShutdownNMT, which is a diagnostic flag and only use for non-production environment, so impact is LOW Likelihood: very reproducible on Solaris Workaround: Don't disable AutoShutdownNMT flag

That said I think the Solaris yield_all implementation may well be obsolete now. And that said needing to yield illustrates a problem with the algorithm. At most it should be used as a scheduling hint as the exact semantics are completely ill-defined in a multiprocessor context. But that aside it may be simplest to just put in a state check and add a transition if needed.

Well it is the os::sleep inside the yield_all (only on Solaris) that requires _thread_in_vm. But looking at the stacktrace why isn't the thread state _thread_in_vm? Does the uncommon trap leave the thread _thread_in_java ?