JDK-7012493 : 6849574/Test.java fails with Internal Error (src/share/vm/prims/jvmtiTagMap.cpp:3294)
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jvmti
  • Affected Version: hs20
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2011-01-14
  • Updated: 2012-02-01
  • Resolved: 2011-03-08
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 6 JDK 7 Other
6u25Fixed 7Fixed hs20Fixed
Related Reports
Relates :  
JDK-6458402 - 3 jvmti tests fail with CMS and +ExplicitGCInvokesConcurrent
Relates :  
JDK-6880972 - examine JVM/TI uses of Threads::number_of_threads() == 0 & SafepointSynchronize::is_at_safepoint()
Description
The following JT_HS test started failing in the 2011.01.13
RT_Baseline nightly:

    compiler/6849574/Test.java

Here is a snippet of the hs_err file from a Solaris X86 Server
VM -Xmixed config:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/tmp/jprt/P1/B/223720.dcubed/source/src/share/vm/prims/jvmtiTagMap.cpp:3294), pid=24173, tid=2
#  assert(SafepointSynchronize::is_at_safepoint()) failed: must be executed at a safepoint
#
# JRE version: 7.0-b124
# Java VM: Java HotSpot(TM) Server VM (20.0-b06-internal-201101122237.dcubed.7009828_for_hsx_20-fastdebug mixed mode solaris-x86 )
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0807cc00):  JavaThread "Unknown thread" [_thread_in_vm, id=2, stack(0xfc73f000,0xfc78f000)]

Stack: [0xfc73f000,0xfc78f000],  sp=0xfc78e580,  free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1ca1579]  void VMError::report(outputStream*)+0x865;;  void VMError::report(outputStream*)+0x865
V  [libjvm.so+0x1ca2746]  void VMError::report_and_die()+0x55e;;  void VMError::report_and_die()+0x55e
V  [libjvm.so+0xaec46a]  void report_vm_error(const char*,int,const char*,const char*)+0x53a;;  void report_vm_error(const char*,int,const char*,const char*)+0x53a
V  [libjvm.so+0x1398edd]  void JvmtiTagMap::weak_oops_do(BoolObjectClosure*,OopClosure*)+0x69d;;  void JvmtiTagMap::weak_oops_do(BoolObjectClosure*,OopClosure*)+0x69d
V  [libjvm.so+0x10c590a]  void JNIHandles::verify()+0x346;;  void JNIHandles::verify()+0x346
V  [libjvm.so+0x1c10a77]  void Universe::verify(bool,bool,bool)+0x29f;;  void Universe::verify(bool,bool,bool)+0x29f
V  [libjvm.so+0x1b9d09a]  int Threads::create_vm(JavaVMInitArgs*,bool*)+0x2aa;;  int Threads::create_vm(JavaVMInitArgs*,bool*)+0x2aa
V  [libjvm.so+0x100aeaf]  JNI_CreateJavaVM+0x77;;  JNI_CreateJavaVM+0x77
C  [libjli.so+0x471f]  InitializeJVM+0x103;;  InitializeJVM+0x103
C  [libjli.so+0x271a]  JavaMain+0x5a;;  JavaMain+0x5a
C  [libc.so.1+0xa7b30]  _thr_setup+0x4e;;  _thr_setup+0x4e
C  [libc.so.1+0xa7e20]  __moddi3+0x60;;  _lwp_start+0x0


Here is a link to one of the failing configs:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-13/RT_Baseline/javase/solaris-i586/server/mixed/solaris-i586_javase_server_mixed_JT_HS/analysis.html


This test failed in 13 of 15 RT_Baseline configs for JT_HS;
2 of the configs have not finished executing yet.
It looks like this particular test fails because of the
-XX:+VerifyBeforeGC option. However, since the assertion
failure happens in VM startup, I think it will apply to
any Java program run with the -XX:+VerifyBeforeGC option
and bits where assertions are enabled (fastdebug and jvmg).

See the work around note for how to suppress this assertion
failure.
Comments
EVALUATION http://hg.openjdk.java.net/jdk7/hotspot/hotspot/rev/c1a0ede55d6f
20-01-2011
EVALUATION http://hg.openjdk.java.net/jdk7/hotspot-rt/hotspot/rev/c1a0ede55d6f
19-01-2011
SUGGESTED FIX Context diffs for the proposed fix: $ hg diff src/share/vm/prims/jvmtiTagMap.cpp diff -r 34d64ad817f4 src/share/vm/prims/jvmtiTagMap.cpp --- a/src/share/vm/prims/jvmtiTagMap.cpp Wed Jan 12 13:59:18 2011 -0800 +++ b/src/share/vm/prims/jvmtiTagMap.cpp Fri Jan 14 18:06:31 2011 -0700 @@ -3290,7 +3290,11 @@ void JvmtiTagMap::follow_references(jint void JvmtiTagMap::weak_oops_do(BoolObjectClosure* is_alive, OopClosure* f) { - assert(SafepointSynchronize::is_at_safepoint(), + // No locks during VM bring-up (0 threads) and no safepoints after main + // thread creation and before VMThread creation (1 thread); initial GC + // verification can happen in that window which gets to here. + assert(Threads::number_of_threads() <= 1 || + SafepointSynchronize::is_at_safepoint(), "must be executed at a safepoint"); if (JvmtiEnv::environments_might_exist()) { JvmtiEnvIterator it;
15-01-2011
EVALUATION Yes, we're creating the VM and this is where we are in src/share/vm/runtime/thread.cpp: 3102 HandleMark hm; 3103 3104 { MutexLocker mu(Threads_lock); 3105 Threads::add(main_thread); 3106 } 3107 3108 // Any JVMTI raw monitors entered in onload will transition into 3109 // real raw monitor. VM is setup enough here for raw monitor enter. 3110 JvmtiExport::transition_pending_onload_raw_monitors(); 3111 3112 if (VerifyBeforeGC && 3113 Universe::heap()->total_collections() >= VerifyGCStartAt) { 3114 Universe::heap()->prepare_for_verify(); HERE => 3115 Universe::verify(); // make sure we're starting with a clean slate 3116 } 3117 3118 // Create the VMThread 3119 { TraceTime timer("Start VMThread", TraceStartupTime); 3120 VMThread::create(); 3121 Thread* vmthread = VMThread::vm_thread(); We're calling Universe::verify() on line 3115. Notice that happens after we created the main thread on line 3105 and before we created the VMThread on line 3120. Can't check for a safepoint because we can't have a safepoint without the VMThread. Threads::number_of_threads() == 0 returns false because we just created the main thread. We're in a "no mans" land in the VM bringup world. I did a quick survey of "oops do" functions in the runtime and prims directories and only found one that asserted being called at a safepoint. It seems like there is a general assumption that "oops do" is called from the "right place". JvmtiTagMap::weak_oops_do() was originally JvmtiTagMap::gc_epilogue() and that's where the safepoint assertion came from. In my experience epilogue functions (and prologue functions) often assert that they are called from a safepoint. It seems that I can change the assertion to include: Threads::number_of_threads() <= 1 or I can delete the safepoint assertion and go with how most of the "oops do" functions are written.
15-01-2011
WORK AROUND Use the following option to supress this assertion failure: -XX:SuppressErrorAt=/jvmtiTagMap.cpp:3294
14-01-2011
EVALUATION The JVM/TI implementation often has to live in the early VM bringup world so there's an easy fix for this problem that is used in other parts of JVM/TI. The assertion needs to be modified like this: // no locks during VM bring-up assert(Threads::number_of_threads() == 0 || SafepointSynchronize::is_at_safepoint(), "must be executed at a safepoint");
14-01-2011
EVALUATION The fix for the following bug appears to be the cause of this assertion failure: 6458402 4/3 3 jvmti tests fail with CMS and +ExplicitGCInvokesConcurrent Looks like the fix for 6458402 changed the code path on which the assertion is checked. JNIHandleBlock::weak_oops_do() was modified to add a call to JvmtiTagMap::weak_oops_do(is_alive, f) which checks this assertion. The following stack snippet: V [libjvm.so+0x1398edd] void JvmtiTagMap::weak_oops_do(BoolObjectClosure*,OopClosure*)+0x69d;; void JvmtiTagMap::weak_oops_do(BoolObjectClosure*,OopClosure*)+0x69d V [libjvm.so+0x10c590a] void JNIHandles::verify()+0x346;; void JNIHandles::verify()+0x346 V [libjvm.so+0x1c10a77] void Universe::verify(bool,bool,bool)+0x29f;; void Universe::verify(bool,bool,bool)+0x29f V [libjvm.so+0x1b9d09a] int Threads::create_vm(JavaVMInitArgs*,bool*)+0x2aa;; int Threads::create_vm(JavaVMInitArgs*,bool*)+0x2aa V [libjvm.so+0x100aeaf] JNI_CreateJavaVM+0x77;; JNI_CreateJavaVM+0x77 shows that JNIHandles::verify() can be called very early in VM startup which results in a call to JNIHandleBlock::weak_oops_do() which results in a call to JvmtiTagMap::weak_oops_do().
14-01-2011