United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-7012493 6849574/Test.java fails with Internal Error (src/share/vm/prims/jvmtiTagMap.cpp:3294)
JDK-7012493 : 6849574/Test.java fails with Internal Error (src/share/vm/prims/jvmtiTagMap.cpp:3294)

Details
Type:
Bug
Submit Date:
2011-01-14
Status:
Closed
Updated Date:
2012-02-01
Project Name:
JDK
Resolved Date:
2011-03-08
Component:
hotspot
OS:
generic
Sub-Component:
jvmti
CPU:
generic
Priority:
P2
Resolution:
Fixed
Affected Versions:
hs20
Fixed Versions:
hs20 (b07)

Related Reports
Backport:
Backport:
Relates:
Relates:

Sub Tasks

Description
The following JT_HS test started failing in the 2011.01.13
RT_Baseline nightly:

    compiler/6849574/Test.java

Here is a snippet of the hs_err file from a Solaris X86 Server
VM -Xmixed config:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/tmp/jprt/P1/B/223720.dcubed/source/src/share/vm/prims/jvmtiTagMap.cpp:3294), pid=24173, tid=2
#  assert(SafepointSynchronize::is_at_safepoint()) failed: must be executed at a safepoint
#
# JRE version: 7.0-b124
# Java VM: Java HotSpot(TM) Server VM (20.0-b06-internal-201101122237.dcubed.7009828_for_hsx_20-fastdebug mixed mode solaris-x86 )
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0807cc00):  JavaThread "Unknown thread" [_thread_in_vm, id=2, stack(0xfc73f000,0xfc78f000)]

Stack: [0xfc73f000,0xfc78f000],  sp=0xfc78e580,  free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1ca1579]  void VMError::report(outputStream*)+0x865;;  void VMError::report(outputStream*)+0x865
V  [libjvm.so+0x1ca2746]  void VMError::report_and_die()+0x55e;;  void VMError::report_and_die()+0x55e
V  [libjvm.so+0xaec46a]  void report_vm_error(const char*,int,const char*,const char*)+0x53a;;  void report_vm_error(const char*,int,const char*,const char*)+0x53a
V  [libjvm.so+0x1398edd]  void JvmtiTagMap::weak_oops_do(BoolObjectClosure*,OopClosure*)+0x69d;;  void JvmtiTagMap::weak_oops_do(BoolObjectClosure*,OopClosure*)+0x69d
V  [libjvm.so+0x10c590a]  void JNIHandles::verify()+0x346;;  void JNIHandles::verify()+0x346
V  [libjvm.so+0x1c10a77]  void Universe::verify(bool,bool,bool)+0x29f;;  void Universe::verify(bool,bool,bool)+0x29f
V  [libjvm.so+0x1b9d09a]  int Threads::create_vm(JavaVMInitArgs*,bool*)+0x2aa;;  int Threads::create_vm(JavaVMInitArgs*,bool*)+0x2aa
V  [libjvm.so+0x100aeaf]  JNI_CreateJavaVM+0x77;;  JNI_CreateJavaVM+0x77
C  [libjli.so+0x471f]  InitializeJVM+0x103;;  InitializeJVM+0x103
C  [libjli.so+0x271a]  JavaMain+0x5a;;  JavaMain+0x5a
C  [libc.so.1+0xa7b30]  _thr_setup+0x4e;;  _thr_setup+0x4e
C  [libc.so.1+0xa7e20]  __moddi3+0x60;;  _lwp_start+0x0


Here is a link to one of the failing configs:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-13/RT_Baseline/javase/solaris-i586/server/mixed/solaris-i586_javase_server_mixed_JT_HS/analysis.html


This test failed in 13 of 15 RT_Baseline configs for JT_HS;
2 of the configs have not finished executing yet.
It looks like this particular test fails because of the
-XX:+VerifyBeforeGC option. However, since the assertion
failure happens in VM startup, I think it will apply to
any Java program run with the -XX:+VerifyBeforeGC option
and bits where assertions are enabled (fastdebug and jvmg).

See the work around note for how to suppress this assertion
failure.

                                    

Comments
EVALUATION

The fix for the following bug appears to be the cause of this
assertion failure:

    6458402 4/3 3 jvmti tests fail with CMS and +ExplicitGCInvokesConcurrent

Looks like the fix for 6458402 changed the code path on
which the assertion is checked. JNIHandleBlock::weak_oops_do()
was modified to add a call to JvmtiTagMap::weak_oops_do(is_alive, f)
which checks this assertion.

The following stack snippet:

V  [libjvm.so+0x1398edd]  void JvmtiTagMap::weak_oops_do(BoolObjectClosure*,OopClosure*)+0x69d;;  void JvmtiTagMap::weak_oops_do(BoolObjectClosure*,OopClosure*)+0x69d
V  [libjvm.so+0x10c590a]  void JNIHandles::verify()+0x346;;  void JNIHandles::verify()+0x346
V  [libjvm.so+0x1c10a77]  void Universe::verify(bool,bool,bool)+0x29f;;  void Universe::verify(bool,bool,bool)+0x29f
V  [libjvm.so+0x1b9d09a]  int Threads::create_vm(JavaVMInitArgs*,bool*)+0x2aa;;  int Threads::create_vm(JavaVMInitArgs*,bool*)+0x2aa
V  [libjvm.so+0x100aeaf]  JNI_CreateJavaVM+0x77;;  JNI_CreateJavaVM+0x77

shows that JNIHandles::verify() can be called very early in VM
startup which results in a call to JNIHandleBlock::weak_oops_do()
which results in a call to JvmtiTagMap::weak_oops_do().
                                     
2011-01-14
EVALUATION

The JVM/TI implementation often has to live in the early VM
bringup world so there's an easy fix for this problem that is
used in other parts of JVM/TI. The assertion needs to be
modified like this:

  // no locks during VM bring-up
  assert(Threads::number_of_threads() == 0 ||
         SafepointSynchronize::is_at_safepoint(),
         "must be executed at a safepoint");
                                     
2011-01-14
WORK AROUND

Use the following option to supress this assertion failure:

    -XX:SuppressErrorAt=/jvmtiTagMap.cpp:3294
                                     
2011-01-14
EVALUATION

Yes, we're creating the VM and this is where we are in
src/share/vm/runtime/thread.cpp:

   3102   HandleMark hm;
   3103 
   3104   { MutexLocker mu(Threads_lock);
   3105     Threads::add(main_thread); 
   3106   }
   3107 
   3108   // Any JVMTI raw monitors entered in onload will transition into
   3109   // real raw monitor. VM is setup enough here for raw monitor enter.
   3110   JvmtiExport::transition_pending_onload_raw_monitors();
   3111 
   3112   if (VerifyBeforeGC &&
   3113       Universe::heap()->total_collections() >= VerifyGCStartAt) {
   3114     Universe::heap()->prepare_for_verify();
HERE => 3115     Universe::verify();   // make sure we're starting with a clean slate
   3116   }
   3117 
   3118   // Create the VMThread
   3119   { TraceTime timer("Start VMThread", TraceStartupTime);
   3120     VMThread::create();
   3121     Thread* vmthread = VMThread::vm_thread();

We're calling Universe::verify() on line 3115. Notice that happens
after we created the main thread on line 3105 and before we created
the VMThread on line 3120.

Can't check for a safepoint because we can't have a safepoint
without the VMThread. Threads::number_of_threads() == 0 returns
false because we just created the main thread. We're in a "no mans"
land in the VM bringup world.

I did a quick survey of "oops do" functions in the runtime and
prims directories and only found one that asserted being called at
a safepoint. It seems like there is a general assumption that
"oops do" is called from the "right place".

JvmtiTagMap::weak_oops_do() was originally JvmtiTagMap::gc_epilogue()
and that's where the safepoint assertion came from. In my experience
epilogue functions (and prologue functions) often assert that they
are called from a safepoint.

It seems that I can change the assertion to include:

    Threads::number_of_threads() <= 1

or I can delete the safepoint assertion and go with how most of the
"oops do" functions are written.
                                     
2011-01-15
SUGGESTED FIX

Context diffs for the proposed fix:

 $ hg diff src/share/vm/prims/jvmtiTagMap.cpp
diff -r 34d64ad817f4 src/share/vm/prims/jvmtiTagMap.cpp
--- a/src/share/vm/prims/jvmtiTagMap.cpp        Wed Jan 12 13:59:18 2011 -0800
+++ b/src/share/vm/prims/jvmtiTagMap.cpp        Fri Jan 14 18:06:31 2011 -0700
@@ -3290,7 +3290,11 @@ void JvmtiTagMap::follow_references(jint
 
 
 void JvmtiTagMap::weak_oops_do(BoolObjectClosure* is_alive, OopClosure* f) {
-  assert(SafepointSynchronize::is_at_safepoint(),
+  // No locks during VM bring-up (0 threads) and no safepoints after main
+  // thread creation and before VMThread creation (1 thread); initial GC
+  // verification can happen in that window which gets to here.
+  assert(Threads::number_of_threads() <= 1 ||
+         SafepointSynchronize::is_at_safepoint(),
          "must be executed at a safepoint");
   if (JvmtiEnv::environments_might_exist()) {
     JvmtiEnvIterator it;
                                     
2011-01-15
EVALUATION

http://hg.openjdk.java.net/jdk7/hotspot-rt/hotspot/rev/c1a0ede55d6f
                                     
2011-01-19
EVALUATION

http://hg.openjdk.java.net/jdk7/hotspot/hotspot/rev/c1a0ede55d6f
                                     
2011-01-20



Hardware and Software, Engineered to Work Together