United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-8016074 NMT: assertion failed: assert(thread->thread_state() == from) failed: coming from wrong thread state
JDK-8016074 : NMT: assertion failed: assert(thread->thread_state() == from) failed: coming from wrong thread state

Details
Type:
Bug
Submit Date:
2013-06-06
Status:
Closed
Updated Date:
2014-01-14
Project Name:
JDK
Resolved Date:
2013-07-04
Component:
hotspot
OS:
solaris
Sub-Component:
runtime
CPU:
sparc
Priority:
P2
Resolution:
Fixed
Affected Versions:
hs24,hs25
Fixed Versions:
hs25 (b41)

Related Reports
Backport:
Backport:
Backport:
Backport:
Backport:

Sub Tasks

Description
Kitchensink/stability test failed due to above assertion failure, with NMT on and -XX:-AutoshutdownNMT

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/java/deployment/zg131198/8011968/hotspot/src/share/vm/runtime/interfaceSupport.hpp:164), pid=8838, tid=18
#  assert(thread->thread_state() == from) failed: coming from wrong thread state
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b76) (build 1.8.0-ea-b76)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b36-internal-debug mixed mode solaris-sparc compressed oops)
# Core dump written. Default location: /tmp/zhgu/ute_run/kitchensink/core or core.8838
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x000000010045b800):  JavaThread "Reference Handler" daemon [_thread_in_Java, id=18, stack(0xffffffff3da00000,0xffffffff3db00000)]

Stack: [0xffffffff3da00000,0xffffffff3db00000],  sp=0xffffffff3dafd250,  free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1883668]  void VMError::report_and_die()+0x7f0
V  [libjvm.so+0xe5eea4]  void report_vm_error(const char*,int,const char*,const char*)+0xcc
V  [libjvm.so+0x8aa068]  void ThreadStateTransition::transition_and_fence(JavaThread*,JavaThreadState,JavaThreadState)+0x80
V  [libjvm.so+0x8a9dd4]  void ThreadStateTransition::trans_and_fence(JavaThreadState,JavaThreadState)+0x34
V  [libjvm.so+0xa5e950]  ThreadBlockInVM::ThreadBlockInVM(JavaThread*)+0x98
V  [libjvm.so+0x158bea4]  int os::sleep(Thread*,long,bool)+0x184
V  [libjvm.so+0x158c268]  void os::yield_all(int)+0x30
V  [libjvm.so+0x1477ef8]  void MemTracker::create_memory_record(unsigned char*,unsigned short,unsigned long,unsigned char*,Thread*)+0x1a0
V  [libjvm.so+0x157bf8c]  void MemTracker::record_malloc(unsigned char*,unsigned long,unsigned short,unsigned char*,Thread*)+0x114
V  [libjvm.so+0x157670c]  void*os::malloc(unsigned long,unsigned short,unsigned char*)+0x384
V  [libjvm.so+0xa09d18]  char*AllocateHeap(unsigned long,unsigned short,unsigned char*,AllocFailStrategy::AllocFailEnum)+0x98
V  [libjvm.so+0xa23274]  void*CHeapObj<(unsigned short)1792>::operator new(unsigned long,unsigned char*)+0x9c
V  [libjvm.so+0xe87d2c]  Deoptimization::UnrollBlock*Deoptimization::fetch_unroll_info_helper(JavaThread*)+0x2c
V  [libjvm.so+0xe8effc]  Deoptimization::UnrollBlock*Deoptimization::uncommon_trap(JavaThread*,int)+0x34
v  blob 0xffffffff730d7a38
J  java.lang.ref.Reference$ReferenceHandler.run()V
v  ~StubRoutines::call_stub
V  [libjvm.so+0x11620e8]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x988
V  [libjvm.so+0x158eacc]  void os::os_exception_wrapper(void(*)(JavaValue*,methodHandle*,JavaCallArguments*,Thread*),JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x44
V  [libjvm.so+0x1161740]  void JavaCalls::call(JavaValue*,methodHandle,JavaCallArguments*,Thread*)+0xd0
V  [libjvm.so+0x11607dc]  void JavaCalls::call_virtual(JavaValue*,KlassHandle,Symbol*,Symbol*,JavaCallArguments*,Thread*)+0x294
V  [libjvm.so+0x1160914]  void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*)+0x9c
V  [libjvm.so+0x1246d00]  void thread_entry(JavaThread*,Thread*)+0x108
V  [libjvm.so+0x17d1014]  void JavaThread::thread_main_inner()+0x1b4
V  [libjvm.so+0x17d0e30]  void JavaThread::run()+0x310
V  [libjvm.so+0x15820d4]  java_start+0x2cc

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  blob 0xffffffff730d7a38
J  java.lang.ref.Reference$ReferenceHandler.run()V
v  ~StubRoutines::call_stub


Apparently, os::yield_all() can only be called by JavaThread when it is in _thread_in_vm state.
                                    

Comments
Well it is the os::sleep inside the yield_all (only on Solaris) that requires _thread_in_vm.

But looking at the stacktrace why isn't the thread state _thread_in_vm? Does the uncommon trap leave the thread _thread_in_java ?
                                     
2013-06-07
That said I think the Solaris yield_all implementation may well be obsolete now.

And that said needing to yield illustrates a problem with the algorithm. At most it should be used as a scheduling hint as the exact semantics are completely ill-defined in a multiprocessor context.

But that aside it may be simplest to just put in a state check and add a transition if needed.
                                     
2013-06-07
Impact: It only impact Solaris with -XX:-AutoShutdownNMT, which is a diagnostic flag and only use for non-production environment, so impact is LOW
Likelihood: very reproducible on Solaris
Workaround: Don't disable AutoShutdownNMT flag


                                     
2013-06-07
Solaris yield_all() implementation has limitations, which do not exist on other platforms. The implementation depends on os::sleep(), which means that it needs JVM Thread implementation, and it blocks on Threads_lock that may cause lock out of order.
                                     
2013-06-26
Yielding NMT worker thread can be one of solution to OOM problem. Raise this priority as part of OOM fix. 
                                     
2013-07-02
URL:   http://hg.openjdk.java.net/hsx/hotspot-rt/hotspot/rev/5f7a4367c787
User:  zgu
Date:  2013-07-04 20:07:15 +0000

                                     
2013-07-04
7u40-critical-request justification: 
  Without the fix, AutoShutdownNMT=false does not work on Solaris, as the flag is one of solutions to OOM problem with Kitchensink tests.

  The risk is low. It redirects the yield call to system thr_yield() on Solaris, the behaviors remain the same on other platforms.

  The fix was reviewed by Karen Kinnear, Coleen Phillimore and Harold Seigel
                                     
2013-07-05
Need SQE-OK prior to approving this.
                                     
2013-07-05
SQE is OK with this backport, it's a low risk fix.
                                     
2013-07-08
URL:   http://hg.openjdk.java.net/hsx/hsx25/hotspot/rev/5f7a4367c787
User:  amurillo
Date:  2013-07-13 03:43:56 +0000

                                     
2013-07-13



Hardware and Software, Engineered to Work Together