JDK-5093457 : CMS: crash in nightly RT_QUICK tests 8/26/04 "Adding a bad referent"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 6
  • Priority: P5
  • Status: Resolved
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2004-08-27
  • Updated: 2004-10-25
  • Resolved: 2004-10-25
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 6
6 b10Fixed
Related Reports
Relates :  
JDK-6563027 - Crash in JVM while running NetBeans 6.0 m9
Relates :  
JDK-4950157 - CMS: vtest died with tiger b26
Description
This is the error log at http://vmsqe.sfbay.sun.com/nightly/mantis/DTWS/results/08-26-04/ClientVM/Solsparc/mixed/Gc_Baseline-Xconc/RT_QUICK-NIGHTLY-Gc_Baseline-Xconc-ClientVM-mixed-Solsparc-2004-08-27-04-50-13/hs_err_pid11266.log

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  Internal Error (/net/prt-solsparc-q1-10/tmp/PrtBuildDir/workspace/src/share/vm/memory/referenceProcessor.cpp, 406 [ Patched ]), pid=11266, tid=8
#
# Java VM: Java HotSpot(TM) Client VM (20040810135029.ysr.merge-debug mixed mode)
#
# Error: assert(referent->is_oop(UseConcMarkSweepGC),"Adding a bad referent")

---------------  T H R E A D  ---------------

Current thread (0x00185518):  ConcurrentMarkSweepThread [id=8]

Stack: [0xf7b00000,0xf7b80000),  sp=0xf7b7f028,  free space=508k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x984b5c]
V  [libjvm.so+0x34c56c]
V  [libjvm.so+0x867df4]
V  [libjvm.so+0x867f10]
V  [libjvm.so+0x866f50]
V  [libjvm.so+0x3283b0]
V  [libjvm.so+0x32503c]
V  [libjvm.so+0x324aec]
V  [libjvm.so+0x32ae68]
V  [libjvm.so+0x32b5b0]
V  [libjvm.so+0x31a8d0]
V  [libjvm.so+0x338a04]
V  [libjvm.so+0x820680]


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00242f50 JavaThread "Thread-6" [_thread_blocked, id=26]
  0x00247458 JavaThread "Thread-5" [_thread_blocked, id=25]
  0x0024e6d0 JavaThread "Thread-2" [_thread_blocked, id=19]
  0x0024d500 JavaThread "Thread-1" [_thread_in_native, id=18]
  0x00050008 JavaThread "DestroyJavaVM" [_thread_blocked, id=1]
  0x0023f678 JavaThread "Thread-0" [_thread_blocked, id=17]
  0x001fd018 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=15]
  0x001fb0e8 JavaThread "CompilerThread0" daemon [_thread_blocked, id=14]
  0x001f7d40 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=13]
  0x001f6cb0 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=12]
  0x001e7b70 JavaThread "Finalizer" daemon [_thread_blocked, id=11]
  0x001e6b38 JavaThread "Reference Handler" daemon [_thread_blocked, id=10]

Other Threads:
  0x001e2fa8 VMThread [id=9]
  0x00218d98 WatcherThread [id=16]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x0004e7d0/0x0004e838] Threads_lock - owner thread: 0x00185518
[0x0004f150/0x0004f1a8] Heap_lock - owner thread: 0x00185518

Heap
 par new generation   total 4032K, used 384K [0xef800000, 0xefc00000, 0xefc00000)
  eden space 3968K,   9% used [0xef800000, 0xef860028, 0xefbe0000)
  from space 64K,   0% used [0xefbe0000, 0xefbe0000, 0xefbf0000)
  to   space 64K,   0% used [0xefbf0000, 0xefbf0000, 0xefc00000)
 concurrent mark-sweep generation total 12288K, used 847K [0xefc00000, 0xf0800000, 0xf3800000)
 concurrent-mark-sweep perm gen total 9360K, used 6671K [0xf3800000, 0xf4124000, 0xf7800000)

Dynamic libraries:
0x00010000 	/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/bin/java
0xff390000 	/lib/libthread.so.1
0xff3b0000 	/lib/libdl.so.1
0xff280000 	/lib/libc.so.1
0xff3f0000 	/platform/SUNW,Ultra-Enterprise/lib/libc_psr.so.1
0xfdc00000 	/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/jre/lib/sparc/client/libjvm.so
0xff240000 	/lib/libsocket.so.1
0xff380000 	/usr/lib/libsched.so.1
0xff200000 	/usr/lib/libCrun.so.1
0xff1e0000 	/lib/libm.so.1
0xfdb00000 	/lib/libnsl.so.1
0xfda00000 	/lib/libm.so.2
0xfdbe0000 	/lib/libmd5.so.1
0xfdbc0000 	/platform/SUNW,Ultra-Enterprise/lib/libmd5_psr.so.1
0xfdae0000 	/lib/libmp.so.2
0xfd9b0000 	/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/jre/lib/sparc/native_threads/libhpi.so
0xfd970000 	/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/jre/lib/sparc/libverify.so
0xfd920000 	/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/jre/lib/sparc/libjava.so
0xfd900000 	/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/jre/lib/sparc/libzip.so
0xfd2b0000 	/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3
0xfaad0000 	/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/jre/lib/sparc/libj2pkcs11.so
0xfd290000 	/usr/lib/libpkcs11.so
0xfaab0000 	/usr/lib/libcryptoutil.so.1
0xfa9e0000 	/lib/libdoor.so.1
0xfa990000 	/usr/lib/security/pkcs11_softtoken.so

VM Arguments:
jvm_args: -DHANGINGJAVA10981 -XX:-PrintVMOptions -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:+ExplicitGCInvokesConcurrent
java_command: runThese -thread 1 -repeat 1 -iter 1 -D:gc=5 -D:TESTBASE=/net/vmsqe.sfbay/export/backup/UNIFIED-DTF/DTWS/suites/RT_QUICK/testbase/tests -runList runList.dat

Environment Variables:
JAVA_HOME=/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc
PATH=/net/vmsqe.sfbay/export/nightly/mantis/JDK/gc_baseline/jdk1.6/solaris-sparc/bin:/usr/bin:/tmp/22878.1.all.q:/usr/local/bin:/usr/ucb:/bin:/usr/bin::/usr/ucb:/net/vmsqe.sfbay/export/backup/UNIFIED-DTF/scripts/tools/solaris-sparc
LD_LIBRARY_PATH=/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/jre/lib/sparc/client:/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/jre/lib/sparc:/var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/jre/../lib/sparc:/import/pkg/local/lib
SHELL=/bin/ksh
DISPLAY=vmsqe.sfbay:0.0


---------------  S Y S T E M  ---------------

OS:                             Solaris 10 s10_51 SPARC
           Copyright 2004 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                            Assembled 21 January 2004

uname:SunOS 5.10 s10_51 sun4u  (T2 libthread)
rlimit: STACK 8192k, CORE 50k, NOFILE 65536, AS infinity
load average:5.83 5.53 5.90

CPU:total 6 has_v8, has_v9, has_vis1

Memory: 8k page, physical 8912896k(7257656k free)

vm_info: Java HotSpot(TM) Client VM (20040810135029.ysr.merge) for solaris-sparc, built on Aug 10 2004 14:01:27 by PRT with unknown Workshop:0x550


-------------------------
Here's the appropriate interval excerpted from the log file:

[Enter:X]
Test c0662202 Passed
#Passed; X; (1,1,0,0,0)
[Exit:X]
......................................
[Enter:X]
Test c0681301 Passed
#Passed; X; (1,1,0,0,0)
[Exit:X]
......................................
[Enter:X]
Test c0685101 Passed
#Passed; X; (1,1,0,0,0)
[Exit:X]
......................................
[Enter:X]
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/referenceProcessor.cpp:406]
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  Internal Error (/net/prt-solsparc-q1-10/tmp/PrtBuildDir/workspace/src/share/vm/memory/referenceProcessor.cpp, 406 [ Patched ]), pid=11266, tid=8
#
# Java VM: Java HotSpot(TM) Client VM (20040810135029.ysr.merge-debug mixed mode)
#
# Error: assert(referent->is_oop(UseConcMarkSweepGC),"Adding a bad referent")
# An error report file with more information is saved as hs_err_pid11266.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#
Current thread is 8
Dumping core ...
/net/vmsqe.sfbay/export/backup/UNIFIED-DTF/DTWS/suites/RT_QUICK/testbase/doit[587]: 11266 Abort
......................................
[Enter:X]
Test c0730101 Passed
#Passed; X; (1,1,0,0,0)
[Exit:X]
......................................
[Enter:X]
Test c0730102 Passed
#Passed; X; (1,1,0,0,0)
[Exit:X]
......................................
[Enter:X]
Test c0741101 Passed
#Passed; X; (1,1,0,0,0)
[Exit:X]
......................................

Here's the runList.crash:

# Fri Aug 27 04:52:06 PDT 2004
$TESTBASE/modena/classes/modena3.2_lang;$TESTBASE/modena/classes/modena3.2_lang/conform/c07_2/c0720101 X

And here's the run report:

MACHINE        : SunOS toasteroven 5.10 s10_51 sun4u sparc SUNW,Ultra-Enterprise
PLATFORM       : solaris-sparc
COMMAND LINE   : ksh /net/vmsqe.sfbay/export/backup/UNIFIED-DTF/DTWS/suites/RT_QUICK/testbase/doit -vm /var/tmp/Work/JDK/NIGHTLY/Gc_Baseline-Xconc/solaris-sparc/bin/java -vmopts -client -DHANGINGJAVA10981 -XX:-PrintVMOptions -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:+ExplicitGCInvokesConcurrent -quick
USER           : 
START TIME     : Fri Aug 27 04:51:54 PDT 2004
END TIME       : Fri Aug 27 05:44:03 PDT 2004
ELAPSED TIME   : 52 Minutes 9 Seconds

Summary Report
---------------------
TEST PASS  : 7916 99% rate
TEST FAIL  : 1 0% rate
TEST RUN   : 7917
---------------------
TEST CRASH : 1 0% rate


I have not done any archeology to figure out when this test first started failing.
Comments
CONVERTED DATA BugTraq+ Release Management Values COMMIT TO FIX: mustang FIXED IN: mustang
02-10-2004
EVALUATION Investigate only after putting back the fixes for 4756801 (and may be 4926388?). Too late to fix for Tiger (not known if a Tiger problem, error never shopwed up in Tiger testing and shows up intermittently in Mustang testing). Committing to Mustang pending further investigation. ###@###.### 2004-10-01: yes this assertion is too strong in Tiger as well, but is less likely without the new Mustang feature of +ExplicitGCInvokesConcurrent. I am not planning to backport this assertion weakening to Tiger since customers will never encounter it in production, and it's unlikely that CTE will encounter it in practice even if using the debug build. I am sropping the priority value to reflect this new assesment of the perceived severity of the bug.
02-10-2004
SUGGESTED FIX for diffs see: http://analemma.sfbay/net/spot/scratch/ysr/critical/webrev.referent Fix putback to gc_baseline on 9/29/04: Event: putback-to Parent workspace: /net/jano.sfbay/export/disk05/hotspot/ws/main/gc_baseline (jano.sfbay:/export/disk05/hotspot/ws/main/gc_baseline) Child workspace: /prt-workspaces/20040928174438.ysr.critical/workspace (prt-web:/prt-workspaces/20040928174438.ysr.critical/workspace) User: ysr Comment: --------------------------------------------------------- Original workspace: neeraja:/net/spot/scratch/ysr/critical Submitter: ysr Archived data: /net/prt-archiver.sfbay/data/archived_workspaces/main/gc_baseline/2004/20040928174438.ysr.critical/ Webrev: http://analemma.sfbay.sun.com/net/prt-archiver.sfbay/data/archived_workspaces/main/gc_baseline/2004/20040928174438.ysr.critical/workspace/webrevs/webrev-2004.09.29/index.html Fixed 5093457: CMS: crash in nightly RT_QUICK tests "Adding a bad referent" http://analemma.sfbay/net/spot/scratch/ysr/critical/webrev.referent Suitably weakened an assertion that was too strong for the CMS case, and updated documentation comment describing the assertion. Also fixed an unrelated (and otherwise harmless) PSScavenge ReferenceProcessor initialization issue. Fix Verified: yes Verification Testing: Jon's BubbleUpRef.java program (test case attached to bug 4950157) with CMS fails within 5 minutes without the fix, passes with the fix: java_g -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent -Xmx64m -XX:+PrintGCDetails BubbleUpRef 16000 50 10000 Reviewed by: John Coomes Testing: . spec, PRT, runThese, BubbleUpRef Files: update: src/share/vm/gc_implementation/parallelScavenge/psScavenge.cpp update: src/share/vm/memory/referenceProcessor.cpp update: src/share/vm/oops/oop.hpp update: src/share/vm/oops/oop.inline.hpp Examined files: 3153 Contents Summary: 4 update 3149 no action (unchanged)
02-10-2004