Bug ID: JDK-8016074 NMT: assertion failed: assert(thread->thread_state() == from) failed: coming from wrong thread state

Type: Bug
Component: hotspot
Sub-Component: runtime
Affected Version: hs24,hs25

Priority: P2
Status: Closed
Resolution: Fixed
OS: solaris
CPU: sparc

Submitted: 2013-06-06
Updated: 2014-01-14
Resolved: 2013-07-04

JDK 7	JDK 8	Other
7u40Fixed	8Fixed	hs24Fixed

Kitchensink/stability test failed due to above assertion failure, with NMT on and -XX:-AutoshutdownNMT

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/java/deployment/zg131198/8011968/hotspot/src/share/vm/runtime/interfaceSupport.hpp:164), pid=8838, tid=18
#  assert(thread->thread_state() == from) failed: coming from wrong thread state
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b76) (build 1.8.0-ea-b76)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b36-internal-debug mixed mode solaris-sparc compressed oops)
# Core dump written. Default location: /tmp/zhgu/ute_run/kitchensink/core or core.8838
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x000000010045b800):  JavaThread "Reference Handler" daemon [_thread_in_Java, id=18, stack(0xffffffff3da00000,0xffffffff3db00000)]

Stack: [0xffffffff3da00000,0xffffffff3db00000],  sp=0xffffffff3dafd250,  free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1883668]  void VMError::report_and_die()+0x7f0
V  [libjvm.so+0xe5eea4]  void report_vm_error(const char*,int,const char*,const char*)+0xcc
V  [libjvm.so+0x8aa068]  void ThreadStateTransition::transition_and_fence(JavaThread*,JavaThreadState,JavaThreadState)+0x80
V  [libjvm.so+0x8a9dd4]  void ThreadStateTransition::trans_and_fence(JavaThreadState,JavaThreadState)+0x34
V  [libjvm.so+0xa5e950]  ThreadBlockInVM::ThreadBlockInVM(JavaThread*)+0x98
V  [libjvm.so+0x158bea4]  int os::sleep(Thread*,long,bool)+0x184
V  [libjvm.so+0x158c268]  void os::yield_all(int)+0x30
V  [libjvm.so+0x1477ef8]  void MemTracker::create_memory_record(unsigned char*,unsigned short,unsigned long,unsigned char*,Thread*)+0x1a0
V  [libjvm.so+0x157bf8c]  void MemTracker::record_malloc(unsigned char*,unsigned long,unsigned short,unsigned char*,Thread*)+0x114
V  [libjvm.so+0x157670c]  void*os::malloc(unsigned long,unsigned short,unsigned char*)+0x384
V  [libjvm.so+0xa09d18]  char*AllocateHeap(unsigned long,unsigned short,unsigned char*,AllocFailStrategy::AllocFailEnum)+0x98
V  [libjvm.so+0xa23274]  void*CHeapObj<(unsigned short)1792>::operator new(unsigned long,unsigned char*)+0x9c
V  [libjvm.so+0xe87d2c]  Deoptimization::UnrollBlock*Deoptimization::fetch_unroll_info_helper(JavaThread*)+0x2c
V  [libjvm.so+0xe8effc]  Deoptimization::UnrollBlock*Deoptimization::uncommon_trap(JavaThread*,int)+0x34
v  blob 0xffffffff730d7a38
J  java.lang.ref.Reference$ReferenceHandler.run()V
v  ~StubRoutines::call_stub
V  [libjvm.so+0x11620e8]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x988
V  [libjvm.so+0x158eacc]  void os::os_exception_wrapper(void(*)(JavaValue*,methodHandle*,JavaCallArguments*,Thread*),JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x44
V  [libjvm.so+0x1161740]  void JavaCalls::call(JavaValue*,methodHandle,JavaCallArguments*,Thread*)+0xd0
V  [libjvm.so+0x11607dc]  void JavaCalls::call_virtual(JavaValue*,KlassHandle,Symbol*,Symbol*,JavaCallArguments*,Thread*)+0x294
V  [libjvm.so+0x1160914]  void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*)+0x9c
V  [libjvm.so+0x1246d00]  void thread_entry(JavaThread*,Thread*)+0x108
V  [libjvm.so+0x17d1014]  void JavaThread::thread_main_inner()+0x1b4
V  [libjvm.so+0x17d0e30]  void JavaThread::run()+0x310
V  [libjvm.so+0x15820d4]  java_start+0x2cc

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  blob 0xffffffff730d7a38
J  java.lang.ref.Reference$ReferenceHandler.run()V
v  ~StubRoutines::call_stub


Apparently, os::yield_all() can only be called by JavaThread when it is in _thread_in_vm state.

SQE is OK with this backport, it's a low risk fix.
08-07-2013
Need SQE-OK prior to approving this.
05-07-2013
7u40-critical-request justification: Without the fix, AutoShutdownNMT=false does not work on Solaris, as the flag is one of solutions to OOM problem with Kitchensink tests. The risk is low. It redirects the yield call to system thr_yield() on Solaris, the behaviors remain the same on other platforms. The fix was reviewed by Karen Kinnear, Coleen Phillimore and Harold Seigel
05-07-2013
Yielding NMT worker thread can be one of solution to OOM problem. Raise this priority as part of OOM fix.
02-07-2013
Solaris yield_all() implementation has limitations, which do not exist on other platforms. The implementation depends on os::sleep(), which means that it needs JVM Thread implementation, and it blocks on Threads_lock that may cause lock out of order.
26-06-2013
Impact: It only impact Solaris with -XX:-AutoShutdownNMT, which is a diagnostic flag and only use for non-production environment, so impact is LOW Likelihood: very reproducible on Solaris Workaround: Don't disable AutoShutdownNMT flag
07-06-2013
That said I think the Solaris yield_all implementation may well be obsolete now. And that said needing to yield illustrates a problem with the algorithm. At most it should be used as a scheduling hint as the exact semantics are completely ill-defined in a multiprocessor context. But that aside it may be simplest to just put in a state check and add a transition if needed.
07-06-2013
Well it is the os::sleep inside the yield_all (only on Solaris) that requires _thread_in_vm. But looking at the stacktrace why isn't the thread state _thread_in_vm? Does the uncommon trap leave the thread _thread_in_java ?
07-06-2013