JDK-7004258 : java vm crash in custom hashmap compiled code
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 6
  • Priority: P4
  • Status: Closed
  • Resolution: Duplicate
  • OS: linux_redhat_5.0
  • CPU: x86
  • Submitted: 2010-12-02
  • Updated: 2012-03-20
  • Resolved: 2011-03-01
Related Reports
Duplicate :  
JDK-6965570 - assert(!needs_patching && x->is_loaded(),"how do we know it's volatile if it's not loaded")
Description
FULL PRODUCT VERSION :
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing)


ADDITIONAL OS VERSION INFORMATION :
Linux idbi05a.sysint.local 2.6.18-128.el5 #1 SMP Wed Dec 17 11:42:39 EST 2008 i686 athlon i386 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
Our standalone oc4j application crashes randomly after we upgraded from jdk1.5.0_11.  OC4J version is 10.1.3.1.0.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The issue is not reproducible.  oc4j just crashes randomly, sometimes after hours and sometimes after days.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Oc4j should not crash.
ACTUAL -
Oc4j crash details are not present as the jvm itself seems to terminate.  There is an hs_err_pid* log generated after each crash.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xb610629e, pid=17345, tid=3021843344
#
# Java VM: Java HotSpot(TM) Client VM (11.3-b02 mixed mode, sharing linux-x86)
# Problematic frame:
# J  EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$HashIterator.hasNext()Z
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x08e74c00):  JavaThread "TaskManager" [_thread_in_Java, id=17399, stack(0xb418a000,0xb41db000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0xec0ac857

Registers:
EAX=0x6c0ac858, EBX=0x8dd83660, ECX=0x6c139418, EDX=0x8dd83c60
ESP=0xb41d9de0, EBP=0xb41d9e08, ESI=0x6c0ac858, EDI=0x0000001c
EIP=0xb610629e, CR2=0xec0ac857, EFLAGS=0x00210206

  Top of Stack: (sp=0xb41d9de0)
0xb41d9de0:   b5e96fcd 6f0e5e30 6c139418 6c139418
0xb41d9df0:   b41d9df0 8dd82814 b41d9e18 8dd82d60
0xb41d9e00:   00000000 8dd82830 b41d9e44 b5e9727d
0xb41d9e10:   8dd83c48 b5e9727d 6c139418 00000001
0xb41d9e20:   6f0e5e30 b41d9e1c 8d73159d b41d9e5c
0xb41d9e30:   8d7abf88 00000000 8d731610 b41d9e18
0xb41d9e40:   b41d9e5c b41d9e88 b5e96fcd 00000000
0xb41d9e50:   00000000 6c139418 6f0e5e30 6f0e13c8

Instructions: (pc=0xb610629e)
0xb610628e:   66 90 8b 71 10 83 fe 00 0f 84 67 00 00 00 3b 06
0xb610629e:   8b be ff ff ff 7f 83 ff 00 0f 85 a0 00 00 00 3b

Stack: [0xb418a000,0xb41db000],  sp=0xb41d9de0,  free space=319k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J  EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$HashIterator.hasNext()Z
j  com.evermind.server.http.HttpApplication.removeUnusedCacheFiles()V+25
j  com.evermind.server.http.HttpSite.removeUnusedCacheFiles()V+31
J  com.evermind.server.http.SessionTimeoutTask.run()V
J  com.evermind.util.Task.schedule(Lcom/evermind/util/TaskManager;)V
J  com.evermind.util.TaskManager.run()V
j  com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run()V+24
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x21b25d]
V  [libjvm.so+0x30f998]
V  [libjvm.so+0x21ab70]
V  [libjvm.so+0x21abfd]
V  [libjvm.so+0x28b265]
V  [libjvm.so+0x3a03dd]
V  [libjvm.so+0x310ac9]
C  [libpthread.so.0+0x549b]


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x08767800 JavaThread "HTTPThreadGroup-5" [_thread_blocked, id=18785, stack(0xb50df000,0xb5130000)]
  0x084d1800 JavaThread "Timer-10" [_thread_blocked, id=17733, stack(0xb48a1000,0xb48f2000)]
  0x082db800 JavaThread "DestroyJavaVM" [_thread_blocked, id=17347, stack(0xb7f13000,0xb7f64000)]
=>0x08e74c00 JavaThread "TaskManager" [_thread_in_Java, id=17399, stack(0xb418a000,0xb41db000)]
  0x08518c00 JavaThread "DefaultQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=17398, stack(0xb41db000,0xb422c000)]
  0x08517400 JavaThread "DefaultQuartzScheduler_Worker-9" [_thread_blocked, id=17397, stack(0xb422c000,0xb427d000)]
  0x08515c00 JavaThread "DefaultQuartzScheduler_Worker-8" [_thread_blocked, id=17396, stack(0xb427d000,0xb42ce000)]
  0x08e6ec00 JavaThread "DefaultQuartzScheduler_Worker-7" [_thread_blocked, id=17395, stack(0xb42ce000,0xb431f000)]
  0x08e6d400 JavaThread "DefaultQuartzScheduler_Worker-6" [_thread_blocked, id=17394, stack(0xb431f000,0xb4370000)]
  0x08e6bc00 JavaThread "DefaultQuartzScheduler_Worker-5" [_thread_blocked, id=17393, stack(0xb4370000,0xb43c1000)]
  0x0860d000 JavaThread "DefaultQuartzScheduler_Worker-4" [_thread_blocked, id=17392, stack(0xb43c1000,0xb4412000)]
  0x0860b800 JavaThread "DefaultQuartzScheduler_Worker-3" [_thread_blocked, id=17391, stack(0xb4412000,0xb4463000)]
  0x08dbd400 JavaThread "DefaultQuartzScheduler_Worker-2" [_thread_blocked, id=17390, stack(0xb4463000,0xb44b4000)]
  0x08549800 JavaThread "DefaultQuartzScheduler_Worker-1" [_thread_blocked, id=17389, stack(0xb44b4000,0xb4505000)]
  0x08dc3800 JavaThread "DefaultQuartzScheduler_Worker-0" [_thread_blocked, id=17388, stack(0xb4505000,0xb4556000)]
  0x08991400 JavaThread "Timer-9" [_thread_blocked, id=17387, stack(0xb47dd000,0xb482e000)]
  0x0898d800 JavaThread "Timer-8" [_thread_blocked, id=17386, stack(0xb482e000,0xb487f000)]
  0x08b1c400 JavaThread "SystemThreadGroup-6" [_thread_in_native, id=17385, stack(0xb4925000,0xb4976000)]
  0x08b1c000 JavaThread "SystemThreadGroup-5" [_thread_in_native, id=17384, stack(0xb4976000,0xb49c7000)]
  0x084dfc00 JavaThread "SystemThreadGroup-4" [_thread_blocked, id=17383, stack(0xb4ad2000,0xb4b23000)]
  0x08b09000 JavaThread "Timer-7" [_thread_blocked, id=17382, stack(0xb49df000,0xb4a30000)]
  0x08af9400 JavaThread "Timer-6" [_thread_blocked, id=17381, stack(0xb4a30000,0xb4a81000)]
  0x08af7c00 JavaThread "Timer-5" [_thread_blocked, id=17380, stack(0xb4a81000,0xb4ad2000)]
  0x08af7000 JavaThread "Timer-4" [_thread_blocked, id=17379, stack(0xb4bc5000,0xb4c16000)]
  0x085e9c00 JavaThread "Timer-3" [_thread_blocked, id=17377, stack(0xb4b23000,0xb4b74000)]
  0x084e2000 JavaThread "Timer-2" [_thread_blocked, id=17376, stack(0xb4b74000,0xb4bc5000)]
  0x08a8cc00 JavaThread "WorkExecutorWorkerThread-1" daemon [_thread_blocked, id=17374, stack(0xb4c6a000,0xb4cbb000)]
  0x085a9000 JavaThread "Thread-8" daemon [_thread_blocked, id=17373, stack(0xb4c19000,0xb4c6a000)]
  0x08a67800 JavaThread "Timer-0" [_thread_blocked, id=17371, stack(0xb4cbb000,0xb4d0c000)]
  0x0888e400 JavaThread "RMIServer [/0.0.0.0:23943] count:1" [_thread_in_native, id=17368, stack(0xb4d1f000,0xb4d70000)]
  0x08980c00 JavaThread "RMIServer [/0.0.0.0:23791] count:1" [_thread_in_native, id=17367, stack(0xb4d70000,0xb4dc1000)]
  0x084c6800 JavaThread "JMSServer[idbi05a.sysint.local:9127]" [_thread_in_native, id=17366, stack(0xb4dcb000,0xb4e1c000)]
  0x088fa000 JavaThread "WsMgmtWorkScheduler" daemon [_thread_blocked, id=17365, stack(0xb4e57000,0xb4ea8000)]
  0x08934800 JavaThread "WsMgmtWorkScheduler" daemon [_thread_blocked, id=17364, stack(0xb4ea8000,0xb4ef9000)]
  0x0891cc00 JavaThread "Timer ServiceThread" [_thread_blocked, id=17363, stack(0xb4ef9000,0xb4f4a000)]
  0x08903000 JavaThread "Scheduler ServiceThread" [_thread_blocked, id=17362, stack(0xb4f4a000,0xb4f9b000)]
  0x088ed800 JavaThread "Event ServiceThread" [_thread_blocked, id=17361, stack(0xb4f9b000,0xb4fec000)]
  0x088cd400 JavaThread "LogFlusher" daemon [_thread_blocked, id=17359, stack(0xb4fec000,0xb503d000)]
  0x088bd000 JavaThread "LogFlusher" daemon [_thread_blocked, id=17358, stack(0xb503d000,0xb508e000)]
  0x088a7000 JavaThread "LogFlusher" daemon [_thread_blocked, id=17357, stack(0xb508e000,0xb50df000)]
  0x083d2800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=17354, stack(0xb5496000,0xb54e7000)]
  0x08312c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=17353, stack(0xb5749000,0xb579a000)]
  0x0830fc00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=17352, stack(0xb579a000,0xb581b000)]
  0x0830e000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=17351, stack(0xb581b000,0xb586c000)]
  0x08305c00 JavaThread "Finalizer" daemon [_thread_blocked, id=17350, stack(0xb5a6c000,0xb5abd000)]
  0x08304400 JavaThread "Reference Handler" daemon [_thread_blocked, id=17349, stack(0xb5abd000,0xb5b0e000)]

Other Threads:
  0x08302800 VMThread [stack: 0xb5b0e000,0xb5b8f000] [id=17348]
  0x083dc400 WatcherThread [stack: 0xb5415000,0xb5496000] [id=17355]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 def new generation   total 1408K, used 790K [0x6c090000, 0x6c210000, 0x6e7f0000)
  eden space 1280K,  52% used [0x6c090000, 0x6c139440, 0x6c1d0000)
  from space 128K,  88% used [0x6c1f0000, 0x6c20c450, 0x6c210000)
  to   space 128K,   0% used [0x6c1d0000, 0x6c1d0000, 0x6c1f0000)
 tenured generation   total 16864K, used 13549K [0x6e7f0000, 0x6f868000, 0x8c090000)
   the space 16864K,  80% used [0x6e7f0000, 0x6f52b470, 0x6f52b600, 0x6f868000)
 compacting perm gen  total 29696K, used 29660K [0x8c090000, 0x8dd90000, 0x94090000)
   the space 29696K,  99% used [0x8c090000, 0x8dd873e8, 0x8dd87400, 0x8dd90000)
    ro space 8192K,  74% used [0x94090000, 0x94688b48, 0x94688c00, 0x94890000)
    rw space 12288K,  58% used [0x94890000, 0x94fa3df0, 0x94fa3e00, 0x95490000)

Dynamic libraries:
00922000-0093c000 r-xp 00000000 fd:00 362000     /lib/ld-2.5.so
0093c000-0093d000 r-xp 00019000 fd:00 362000     /lib/ld-2.5.so
0093d000-0093e000 rwxp 0001a000 fd:00 362000     /lib/ld-2.5.so
00940000-00a7e000 r-xp 00000000 fd:00 362030     /lib/libc-2.5.so
00a7e000-00a80000 r-xp 0013e000 fd:00 362030     /lib/libc-2.5.so
00a80000-00a81000 rwxp 00140000 fd:00 362030     /lib/libc-2.5.so
00a81000-00a84000 rwxp 00a81000 00:00 0
00a86000-00a88000 r-xp 00000000 fd:00 362033     /lib/libdl-2.5.so
00a88000-00a89000 r-xp 00001000 fd:00 362033     /lib/libdl-2.5.so
00a89000-00a8a000 rwxp 00002000 fd:00 362033     /lib/libdl-2.5.so
00a8c000-00ab1000 r-xp 00000000 fd:00 362034     /lib/libm-2.5.so
00ab1000-00ab2000 r-xp 00024000 fd:00 362034     /lib/libm-2.5.so
00ab2000-00ab3000 rwxp 00025000 fd:00 362034     /lib/libm-2.5.so
00ab5000-00ac8000 r-xp 00000000 fd:00 362035     /lib/libpthread-2.5.so
00ac8000-00ac9000 r-xp 00012000 fd:00 362035     /lib/libpthread-2.5.so
00ac9000-00aca000 rwxp 00013000 fd:00 362035     /lib/libpthread-2.5.so
00aca000-00acc000 rwxp 00aca000 00:00 0
00b2b000-00b32000 r-xp 00000000 fd:00 362036     /lib/librt-2.5.so
00b32000-00b33000 r-xp 00006000 fd:00 362036     /lib/librt-2.5.so
00b32000-00b33000 r-xp 00006000 fd:00 362036     /lib/librt-2.5.so
00b33000-00b34000 rwxp 00007000 fd:00 362036     /lib/librt-2.5.so
00bf0000-00bfb000 r-xp 00000000 fd:00 362032     /lib/libgcc_s-4.1.2-20080825.so.1
00bfb000-00bfc000 rwxp 0000a000 fd:00 362032     /lib/libgcc_s-4.1.2-20080825.so.1
00c3f000-00c52000 r-xp 00000000 fd:00 362042     /lib/libnsl-2.5.so
00c52000-00c53000 r-xp 00012000 fd:00 362042     /lib/libnsl-2.5.so
00c53000-00c54000 rwxp 00013000 fd:00 362042     /lib/libnsl-2.5.so
00c54000-00c56000 rwxp 00c54000 00:00 0
00c58000-00d38000 r-xp 00000000 fd:03 428559     /usr/lib/libstdc++.so.6.0.8
00d38000-00d3c000 r-xp 000df000 fd:03 428559     /usr/lib/libstdc++.so.6.0.8
00d3c000-00d3d000 rwxp 000e3000 fd:03 428559     /usr/lib/libstdc++.so.6.0.8
00d3d000-00d43000 rwxp 00d3d000 00:00 0
06000000-0642a000 r-xp 00000000 fd:03 296991     /usr/java/jdk1.6.0_13/jre/lib/i386/client/libjvm.so
0642a000-06444000 rwxp 0042a000 fd:03 296991     /usr/java/jdk1.6.0_13/jre/lib/i386/client/libjvm.so
06444000-06864000 rwxp 06444000 00:00 0
08048000-08052000 r-xp 00000000 fd:03 294940     /usr/java/jdk1.6.0_13/bin/java
08052000-08053000 rwxp 00009000 fd:03 294940     /usr/java/jdk1.6.0_13/bin/java
082d7000-08f44000 rwxp 082d7000 00:00 0          [heap]
6c090000-6c210000 rwxp 6c090000 00:00 0
6c210000-6e7f0000 rwxp 6c210000 00:00 0
6e7f0000-6f868000 rwxp 6e7f0000 00:00 0
6f868000-8c090000 rwxp 6f868000 00:00 0
8c090000-8dd90000 rwxp 8c090000 00:00 0
8dd90000-94090000 rwxp 8dd90000 00:00 0
94090000-94689000 r-xs 00001000 fd:03 296853     /usr/java/jdk1.6.0_13/jre/lib/i386/client/classes.jsa
94689000-94890000 rwxp 94689000 00:00 0
94890000-94fa4000 rwxp 005fa000 fd:03 296853     /usr/java/jdk1.6.0_13/jre/lib/i386/client/classes.jsa
94fa4000-95490000 rwxp 94fa4000 00:00 0
95490000-9556c000 rwxp 00d0e000 fd:03 296853     /usr/java/jdk1.6.0_13/jre/lib/i386/client/classes.jsa
9556c000-95890000 rwxp 9556c000 00:00 0
95890000-95894000 r-xs 00dea000 fd:03 296853     /usr/java/jdk1.6.0_13/jre/lib/i386/client/classes.jsa
95894000-95c90000 rwxp 95894000 00:00 0
b418a000-b418d000 --xp b418a000 00:00 0
....................
....................

VM Arguments:
jvm_args: -Djava.library.path=/u001/app/oracle/product/10.1.3/OracleBI/server/Bin:/u001/app/oracle/product/10.1.3/OracleBI/web/bin -Dcom.sun.management.jmxremote -DSAROOTDIR=/u001/app/oracle/product/10.1.3/OracleBI -DSADATADIR=/u001/app/oracle/product/10.1.3/OracleBIData -XX:MaxPermSize=128m -Xmx512m
java_command: /u001/app/oracle/product/10.1.3/OracleBI/oc4j_bi/j2ee/home/oc4j.jar -config /u001/app/oracle/product/10.1.3/OracleBI/oc4j_bi/j2ee/home/config/server.xml -userThreads
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/u001/app/oracle/product/10.2/db/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/mnt/home/mkrishnakumar/bin:/mnt/home/mkrishnakumar/bin
LD_LIBRARY_PATH=/usr/java/jdk1.6.0_13/jre/lib/i386/client:/usr/java/jdk1.6.0_13/jre/lib/i386:/usr/java/jdk1.6.0_13/jre/../lib/i386::/u001/app/oracle/lib
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x3cd180], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x3cd180], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x30ea50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x30ea50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x30ea50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x30ea50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x310f20], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x310cc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x310cc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x310cc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


---------------  S Y S T E M  ---------------

OS:Red Hat Enterprise Linux Server release 5.3 (Tikanga)

uname:Linux 2.6.18-128.el5 #1 SMP Wed Dec 17 11:42:39 EST 2008 i686
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 16384, NOFILE 10240, AS infinity
load average:0.06 0.03 0.00

CPU:total 1 (2 cores per cpu, 1 threads per core) family 15 model 65 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, mmxext, 3dnow, 3dnowext

Memory: 4k page, physical 1035244k(38984k free), swap 2097144k(2097084k free)

vm_info: Java HotSpot(TM) Client VM (11.3-b02) for linux-x86 JRE (1.6.0_13-b03), built on Mar  9 2009 01:29:45 by "java_re" with gcc 3.2.1-7a (J2SE release)

time: Thu Jun 24 10:56:01 2010
elapsed time: 2471 seconds


REPRODUCIBILITY :
This bug can be reproduced often.
Comments
EVALUATION I believe this is a product version of 6965570. It looks like a patching volatile read that didn't get a patching stub. Instructions: (pc=0xb610629e) 0xb610628e: 66 90 8b 71 10 83 fe 00 0f 84 67 00 00 00 3b 06 0xb610629e: 8b be ff ff ff 7f 83 ff 00 0f 85 a0 00 00 00 3b ;; b610628e 66 data16 ;; b610628f 90 nop ;; b6106290 8b 71 10 mov 0x10(%ecx),%esi ;; b6106293 83 fe 00 cmp $0x0,%esi ;; b6106296 0f 84 67 00 00 00 je 0xb6106303 ;; b610629c 3b 06 cmp (%esi),%eax ;; --------------- ;; b610629e 8b be ff ff ff 7f mov 0x7fffffff(%esi),%edi ;; b61062a4 83 ff 00 cmp $0x0,%edi ;; b61062a7 0f 85 a0 00 00 00 jne 0xb610634d ;; b61062ad 3b ff cmp %edi,%edi 0x7fffffff is max_jint which is the placeholder value we used for patching addresses. If 6354181 occurred this is exactly what we'd emit.
25-02-2011
EVALUATION Can the submitter please provide the context of the crash? We need more info to better understand the bug. Timeout in 2-weeks.
16-02-2011
EVALUATION Sorry I see the description has an hs_err file, so all we need to do is know why it crashed.
18-01-2011
EVALUATION We have no way of reproducing this or matching this to another bug. Can the submitter rerun and post the hs_err_pid* file and try using an .hotspot_compiler file with the contents (all one line) with the option -XX:+UnlockDiagnosticVMOptions . option EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$HashIterator.hasNext PrintNMethods Otherwise submit a test case so we can reproduce this crash. Marking this as Needs More Information for a week or so.
18-01-2011