JDK-6399567 : iCMS: JumbleGC002 and LoadUnloadGC2 intermittently crash in nightly testing w/-server
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 6
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • OS: generic,solaris
  • CPU: generic,sparc
  • Submitted: 2006-03-16
  • Updated: 2012-02-01
  • Resolved: 2006-06-21
Related Reports
Duplicate :  
JDK-6415406 - ATG client crash with fastdebug build at methodOopDesc::bci_from
Description
The nightly tests:

gc/gctests/JumbleGC002
gc/gctests/LoadUnloadGC2

are failing with iCMS with -server intermittently.
The latter crashes with higher frequency than the former
and usually displays one of two crash logs:

(1) crash#1 with LoadUnloadGC2:
http://vmsqe.sfbay/nightly/mantis/DTWS/results/03-10-06/ServerVM/Solsparc/mixed/Gc_Baseline-Xinc/vm.gc-NIGHTLY-Gc_Baseline-Xinc-ServerVM-mixed-Solsparc-2006-03-10-23-20-30/ResultDir/LoadUnloadGC2/LoadUnloadGC2.tlog
#annotate TEST javaopt=-server -Xmixed -XX:-PrintVMOptions -Xincgc -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:-UseGCTimeLimit
/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparc/bin/java -server -Xmixed -DHANGINGJAVA10935 -XX:-PrintVMOptions -Xincgc -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:-UseGCTimeLimit gc.gctests.LoadUnloadGC2.LoadUnloadGC2
##Exit status of execution step=6
##Core file exists
##!checkExitCode

## To suppress the following error report, specify this argument
## after -XX: or in .hotspotrc:  SuppressErrorAt=/concurrentMarkSweepGeneration.cpp:5965]
##
## An unexpected error has been detected by Java Runtime Environment:
##
##  Internal Error (/net/prt-solsparc-q1-16/tmp/PrtBuildDir/workspace/src/share/vm/memory/concurrentMarkSweepGeneration.cpp, 5965 [ Patched ]), pid=1470, tid=7
##
## Java VM: Java HotSpot(TM) Server VM (20060307095526.jcoomes.gc_bugs-debug mixed mode)
##
## Error: assert(this_oop->is_oop(),"expected an oop")
## An error report file with more information is saved as hs_err_pid1470.log
##
## If you would like to submit a bug report, please visit:
##   http://java.sun.com/webapps/bugreport/crash.jsp
##
#VM option '-PrintVMOptions'
#VM option '+CMSPermGenSweepingEnabled'
#VM option '+CMSClassUnloadingEnabled'
#VM option '-UseGCTimeLimit'
#Current thread is 7
#Dumping core ...

#Please use -XX:-UseGCOverheadLimit in place of -XX:-UseGCTimeLimit in the future

(3) Crash#1 with JumbleGC002:
http://vmsqe.sfbay/nightly/mantis/DTWS/results/03-10-06/ServerVM/Solsparc/mixed/Gc_Baseline-Xinc/vm.gc-NIGHTLY-Gc_Baseline-Xinc-ServerVM-mixed-Solsparc-2006-03-10-23-20-30/ResultDir/JumbleGC002/JumbleGC002.tlog
## To suppress the following error report, specify this argument
## after -XX: or in .hotspotrc:  SuppressErrorAt=/genOopClosures.inline.hpp:127]
##
## An unexpected error has been detected by Java Runtime Environment:
##
##  Internal Error (/net/prt-solsparc-q1-16/tmp/PrtBuildDir/workspace/src/share/vm/memory/genOopClosures.inline.hpp, 127 [ Patched ]), pid=1133, tid=4
##
## Java VM: Java HotSpot(TM) Server VM (20060307095526.jcoomes.gc_bugs-debug mixed mode)
##
## Error: assert(!_g->to()->is_in_reserved(obj),"Scanning field twice?")
## An error report file with more information is saved as hs_err_pid1133.log
##
## If you would like to submit a bug report, please visit:
##   http://java.sun.com/webapps/bugreport/crash.jsp
##
#VM option '-PrintVMOptions'
#VM option '+CMSPermGenSweepingEnabled'
#VM option '+CMSClassUnloadingEnabled'
#Current thread is 4
#Dumping core ...

An identical crash was observed also on 3/15:
http://vmsqe.sfbay/nightly/mantis/DTWS/results/03-15-06/ServerVM/Solsparc/mixed/Gc_Baseline-Xinc/vm.gc-NIGHTLY-Gc_Baseline-Xinc-ServerVM-mixed-Solsparc-2006-03-15-22-58-33/ResultDir/JumbleGC002/JumbleGC002.tlog

The hs_err file from that incident is below:
http://vmsqe.sfbay.sun.com/nightly/mantis/DTWS/results/03-15-06/ServerVM/Solsparc/mixed/Gc_Baseline-Xinc/vm.gc-NIGHTLY-Gc_Baseline-Xinc-ServerVM-mixed-Solsparc-2006-03-15-22-58-33/hs_err_pid20860.log
#
# An unexpected error has been detected by Java Runtime Environment:
#
#  Internal Error (/net/prt-solsparc-q1-10/tmp/PrtBuildDir/workspace/src/share/vm/memory/genOopClosures.inline.hpp, 127 [ Patched ]), pid=20860, tid=4
#
# Java VM: Java HotSpot(TM) Server VM (20060314120145.ysr.dragon-debug mixed mode)
#
# Error: assert(!_g->to()->is_in_reserved(obj),"Scanning field twice?")
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00051c00):  GCTaskThread [id=4]

Stack: 
[error occurred during error reporting, step 110, id 0xe0000000]

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xf56f18]
V  [libjvm.so+0x3fb4b8]
V  [libjvm.so+0xf55e88]
V  [libjvm.so+0xf56f18]
V  [libjvm.so+0x3fb4b8]
V  [libjvm.so+0x55b6f8]
V  [libjvm.so+0x502c60]
V  [libjvm.so+0xe75eac]
V  [libjvm.so+0xd807dc]
V  [libjvm.so+0x4b6574]
V  [libjvm.so+0xc48954]
V  [libjvm.so+0xf88e2c]
V  [libjvm.so+0xc24908]


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x0029e800 JavaThread "Thread-198" [_thread_blocked, id=214]
  0x0029d400 JavaThread "Thread-197" [_thread_blocked, id=213]
  0x001b9800 JavaThread "Thread-195" [_thread_blocked, id=211]
  0x001b8400 JavaThread "Thread-194" [_thread_blocked, id=210]
  0x001b5c00 JavaThread "Thread-192" [_thread_blocked, id=208]
  0x0011bc00 JavaThread "Thread-189" [_thread_blocked, id=205]
  0x0030c800 JavaThread "Thread-181" [_thread_blocked, id=197]
  0x003b7800 JavaThread "Thread-176" [_thread_blocked, id=192]
  0x004a6000 JavaThread "Thread-171" [_thread_blocked, id=187]
  0x00359400 JavaThread "Thread-161" [_thread_blocked, id=177]
  0x0019fc00 JavaThread "Thread-156" [_thread_blocked, id=172]
  0x0034b000 JavaThread "Thread-152" [_thread_blocked, id=168]
  0x00349800 JavaThread "Thread-151" [_thread_blocked, id=167]
  0x00348400 JavaThread "Thread-150" [_thread_blocked, id=166]
  0x00344000 JavaThread "Thread-147" [_thread_blocked, id=163]
  0x00368400 JavaThread "Thread-145" [_thread_blocked, id=161]
  0x00364800 JavaThread "Thread-142" [_thread_blocked, id=158]
  0x00362000 JavaThread "Thread-140" [_thread_blocked, id=156]
  0x00376000 JavaThread "Thread-138" [_thread_blocked, id=154]
  0x00374800 JavaThread "Thread-137" [_thread_blocked, id=153]
  0x00371c00 JavaThread "Thread-135" [_thread_blocked, id=151]
  0x0036dc00 JavaThread "Thread-132" [_thread_blocked, id=148]
  0x0036b000 JavaThread "Thread-130" [_thread_blocked, id=146]
  0x00360400 JavaThread "Thread-128" [_thread_blocked, id=144]
  0x0035f000 JavaThread "Thread-127" [_thread_blocked, id=143]
  0x00404400 JavaThread "Thread-125" [_thread_blocked, id=141]
  0x00396800 JavaThread "Thread-115" [_thread_blocked, id=131]
  0x00392800 JavaThread "Thread-112" [_thread_blocked, id=128]
  0x00391400 JavaThread "Thread-111" [_thread_blocked, id=127]
  0x0038e400 JavaThread "Thread-109" [_thread_blocked, id=125]
  0x003fe400 JavaThread "Thread-104" [_thread_blocked, id=120]
  0x003fa800 JavaThread "Thread-101" [_thread_blocked, id=117]
  0x003f2c00 JavaThread "Thread-95" [_thread_blocked, id=111]
  0x003f0400 JavaThread "Thread-93" [_thread_blocked, id=109]
  0x003ec800 JavaThread "Thread-90" [_thread_blocked, id=106]
  0x003ea000 JavaThread "Thread-88" [_thread_blocked, id=104]
  0x003e7800 JavaThread "Thread-86" [_thread_blocked, id=102]
  0x003e5000 JavaThread "Thread-84" [_thread_blocked, id=100]
  0x003dd400 JavaThread "Thread-78" [_thread_blocked, id=94]
  0x003d5800 JavaThread "Thread-73" [_thread_blocked, id=89]
  0x003ce800 JavaThread "Thread-68" [_thread_blocked, id=84]
  0x003ca800 JavaThread "Thread-65" [_thread_blocked, id=81]
  0x003c9400 JavaThread "Thread-64" [_thread_blocked, id=80]
  0x003bd000 JavaThread "Thread-55" [_thread_blocked, id=71]
  0x0037b800 JavaThread "Thread-43" [_thread_blocked, id=59]
  0x00389c00 JavaThread "Thread-39" [_thread_blocked, id=55]
  0x00278c00 JavaThread "Thread-33" [_thread_blocked, id=49]
  0x00276400 JavaThread "Thread-31" [_thread_blocked, id=47]
  0x0025a800 JavaThread "Thread-19" [_thread_blocked, id=35]
  0x00258400 JavaThread "Thread-17" [_thread_blocked, id=33]
  0x0024a400 JavaThread "Thread-7" [_thread_blocked, id=23]
  0x001e2c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=15]
  0x001e0800 JavaThread "CompilerThread1" daemon [_thread_blocked, id=14]
  0x001df000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=13]
  0x001dd800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=12]
  0x001da400 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=11]
  0x001c4400 JavaThread "Finalizer" daemon [_thread_blocked, id=10]
  0x001c3400 JavaThread "Reference Handler" daemon [_thread_blocked, id=9]
  0x0004b000 JavaThread "main" [_thread_blocked, id=2]

Other Threads:
  0x001bf800 VMThread [id=8]
  0x001e4c00 WatcherThread [id=16]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00049588/0x000495f8] Threads_lock - owner thread: 0x001bf800
[0x00049f58/0x00049fb8] Heap_lock - owner thread: 0x0024a400

Heap
 par new generation   total 4032K, used 4002K [0xf2800000, 0xf2c00000, 0xf2c00000)
  eden space 3968K, 100% used [0xf2800000, 0xf2be8938, 0xf2be0000)
  from space 64K,   0% used [0xf2bf0000, 0xf2bf0000, 0xf2c00000)
  to   space 64K,   0% used [0xf2be0000, 0xf2be0000, 0xf2bf0000)
 concurrent mark-sweep generation total 61440K, used 15288K [0xf2c00000, 0xf6800000, 0xf6800000)
 concurrent-mark-sweep perm gen total 16384K, used 1690K [0xf6800000, 0xf7800000, 0xfa800000)

Dynamic libraries:
0x00010000 	/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/bin/java
0xff370000 	/usr/lib/libthread.so.1
0xff350000 	/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/bin/../jre/lib/sparc/jli/libjli.so
0xff3fa000 	/usr/lib/libdl.so.1
0xff200000 	/usr/lib/libc.so.1
0xff3e6000 	/usr/platform/SUNW,Ultra-80/lib/libc_psr.so.1
0xfd400000 	/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/jre/lib/sparc/server/libjvm.so
0xff2f0000 	/usr/lib/libsocket.so.1
0xff2d0000 	/usr/lib/libsched.so.1
0xff1c0000 	/usr/lib/libCrun.so.1
0xff170000 	/usr/lib/libm.so.1
0xff150000 	/usr/lib/libdoor.so.1
0xfd300000 	/usr/lib/libnsl.so.1
0xff130000 	/usr/lib/libmp.so.2
0xfd2e0000 	/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/jre/lib/sparc/native_threads/libhpi.so
0xfd290000 	/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/jre/lib/sparc/libverify.so
0xfd1c0000 	/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/jre/lib/sparc/libjava.so
0xfd1a0000 	/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/jre/lib/sparc/libzip.so

VM Arguments:
jvm_args: -Xmixed -DHANGINGJAVA5722 -XX:-PrintVMOptions -Xincgc -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
java_command: gc.gctests.JumbleGC002 -iterations=1 -gcTimeout=5
Launcher Type: SUN_STANDARD

Environment Variables:
CLASSPATH=/net/gtee.sfbay/export/nightly/mustang/DTWS/results/03-15-06/ServerVM/Solsparc/mixed/Gc_Baseline-Xinc/WORK_DIR/vm.gc-NIGHTLY-Gc_Baseline-Xinc-ServerVM-mixed-Solsparc-2006-03-15-22-58-33/run2/fhsu.Solaris.sparc/JumbleGC002:/net/gtee.sfbay/export/gtee/suites/testbase_vm.1.6/vm/bin/classes:/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparc/lib/tools.jar
PATH=/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparc/bin:/bin:/usr/bin:/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparc/jre/bin:/mksnt
LD_LIBRARY_PATH=/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/jre/lib/sparc/server:/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/jre/lib/sparc:/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparcv9/jre/../lib/sparc:/net/gtee.sfbay/export/gtee/suites/testbase_vm.1.6/vm/src/nsk/share/lib/solaris-sparc:/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-sparc/jre/lib/sparc/server:/net/gtee.sfbay/export/gtee/suites/testbase_vm.1.6/vm/bin/lib/solaris-sparc/nsk/share/gc/lock/jni:/net/gtee.sfbay/export/gtee/suites/testbase_vm.1.6/vm/bin/lib/solaris-sparc/nsk/share/gc/lock/malloc:/net/gtee.sfbay/export/gtee/suites/testbase_vm.1.6/vm/bin/lib/solaris-sparc/nsk/share/gc/lock/jvmti:/net/gtee.sfbay/export/gtee/suites/testbase_vm.1.6/vm/bin/lib/solaris-sparc/nsk/share/gc/lock/jniref
SHELL=/usr/bin/sh
DISPLAY=vmsqe.sfbay:0.0

Signal Handlers:
SIGSEGV: [libjvm.so+0xf584cc], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0xf584cc], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0xc2d80c], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0xc2d80c], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0xc2d80c], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGUSR1: [libjvm.so+0xc2d82c], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIGUSR2: [libjvm.so+0xc2d80c], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGHUP: [libjvm.so+0xc27918], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGINT: [libjvm.so+0xc27918], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0xc27918], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0xc27918], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0xc27918], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0xc27918], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGUSR1: [libjvm.so+0xc2d82c], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIGUSR2: [libjvm.so+0xc2d80c], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c


---------------  S Y S T E M  ---------------

OS:                        Solaris 9 9/04 s9s_u7wos_09 SPARC
           Copyright 2004 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                             Assembled 29 June 2004

uname:SunOS 5.9 Generic_117171-07 sun4u  (T2 libthread)
rlimit: STACK 8192k, CORE 50k, NOFILE 65536, AS infinity
load average:5.74 5.29 5.12

CPU:total 4 has_v8, has_v9, has_vis1

Memory: 8k page, physical 2097152k(1135768k free)

vm_info: Java HotSpot(TM) Server VM (20060314120145.ysr.dragon) for solaris-sparc, built on Mar 14 2006 14:40:35 by "PRT" with unknown Workshop:0x580