JDK 1.4.0 client hotspot crashes with following error:
Unexpected Signal : 11 occurred at PC=0xFED0AFC8
Function=[Unknown. Nearest: JVM_GetStackAccessControlContext+0x10B4]
Library=/usr/java1.4.0/jre/lib/sparc/client/libjvm.so
Current Java thread:
at java.lang.Throwable.fillInStackTrace(Native Method)
- locked <764b08f0> (a java.lang.NullPointerException)
at java.lang.Throwable.<init>(Throwable.java:180)
at java.lang.Exception.<init>(Exception.java:29)
at java.lang.RuntimeException.<init>(RuntimeException.java:32)
at
java.lang.NullPointerException.<init>(NullPointerException.java:36)
[error occured during error reporting]
initialize TLAB: thread: 0x00000000 size: 0KB used
Dbx stack is as follows
(complete dbx stack is in attached file hotspot_crash_1006.tar.gz):
t@1988 (l@1988) terminated by signal SEGV (no mapping at the fault address)
0xfedba048: scope_desc_at+0x01a4: ld [%l3 + 0x4], %g2
(dbx)
(dbx) where
current thread: t@1988
=>[1] nmethod::scope_desc_at(0xfb401d48, 0xfb4027a0, 0x0, 0xfb4027a0, 0x203534, 0xfef123dc), at 0xfedba048
[2] vframe::new_vframe(0x7237c9d0, 0x7237c9e0, 0x841810, 0x24, 0x2d6694, 0xfef125a4), at 0xfef12470
[3] vframe::sender(0x466178, 0x466178, 0xfefa9611, 0xfefa960a, 0x4, 0x0), at 0xfef12600
[4] JavaThread::print_stack(0x841810, 0xfef969e8, 0xff3a196c, 0x23ffa8, 0xfeedfcc8, 0x0), at 0xfef0471c
[5] os::report_fatal_error(0xffffffff, 0x0, 0xfef96867, 0x1, 0xfeffa060, 0x7237d880), at 0xfeedfd18
[6] os::handle_unexpected_exception(0x841810, 0xb, 0xfed0afc8, 0x7237dd28, 0xfede6408, 0x0), at 0xfeee0164
[7] JVM_handle_solaris_signal(0xfed0afc8, 0x7237dd28, 0x7237da70, 0xfed0afc8, 0xfed0afcc, 0x0), at 0xfede6c2c
[8] __sighndlr(0xb, 0x7237dd28, 0x7237da70, 0xfede63a4, 0x0, 0x0), at 0xff374ca4
[9] call_user_handler(0x75da4e00, 0x7c4, 0xff3878e0, 0x7237da70, 0x7237dd28, 0xb), at 0xff36fadc
[10] sigacthandler(0x75da4e00, 0x7237dd28, 0x7237da70, 0xff386000, 0x7237dd28, 0xb), at 0xff36fca8
---- called from signal handler with signal 11 (SIGSEGV) ------
[11] vframeStream::fill_from_frame(0x7237df10, 0x7237ea50, 0x1, 0x0, 0x1, 0xfac0d0e4), at 0xfed0afc8
[12] vframeStream::next(0x7237df10, 0xe6844fe0, 0xff01fa6c, 0xe6802e28, 0xff01fa6c, 0x7237e3d8), at 0xfed0b398
[13] java_lang_Throwable::fill_in_stack_trace(0xfeff0198, 0xff022350, 0xff000a0c, 0xfefff8cc, 0x2f2aa4, 0xff001bb4), at 0xfed2a23c
[14] java_lang_Throwable::fill_in_stack_trace(0x7237e170, 0x7237e6a8, 0xa, 0xe6844f50, 0xfac0bc8c, 0x7237e768), at 0xfed29770
[15] JVM_FillInStackTrace(0x84189c, 0x7237e2cc, 0x7237e764, 0x841810, 0x841810, 0x0), at 0xfed295b8
[16] Java_java_lang_Throwable_fillInStackTrace(0x84189c, 0x7237e2cc, 0x7237e690, 0x7237e68c, 0x7237e764, 0x0), at 0xfebac10c
[17] 0xfac0d134(0x764b08f0, 0x7237e340, 0x7237e344, 0x7237e740, 0x0, 0x7237e268), at 0xfac0d133
[18] 0xfac05c4c(0x764b08f0, 0x7237e3b8, 0x7237e3bc, 0xfac15410, 0x7237e964, 0x7237e2e8), at 0xfac05c4b
[19] 0xfac05da0(0x764b08f0, 0x7237e430, 0x7237e434, 0xfac155dc, 0x2ba30, 0x7237e360), at 0xfac05d9f
[20] 0xfac05da0(0x764b08f0, 0x7237e4a8, 0x7237e4ac, 0xfac155dc, 0x841810, 0x7237e3d8), at 0xfac05d9f
[21] 0xfac05da0(0x841810, 0x0, 0x0, 0xfac155dc, 0x2f2aa4, 0x7237e450), at 0xfac05d9f
[22] 0xfac00438(0x7237e538, 0x7237e6a8, 0xa, 0xe6844f50, 0xfac0bc8c, 0x7237e768), at 0xfac00437
[23] JavaCalls::call_helper(0x7237e6a0, 0x7237e5f4, 0x7237e764, 0x841810, 0x841810, 0x0), at 0xfecfd9a4
[24] JavaCalls::call_special(0x7237e6a0, 0x7237e698, 0x7237e690, 0x7237e68c, 0x7237e764, 0x841810), at 0xfed06f68
[25] Exceptions::new_exception(0x7237e974, 0x841810, 0x7237e744, 0x7237e740, 0x7237e764, 0x7237e73c), at 0xfed318a8
[26] Exceptions::new_exception(0x7237e974, 0x841810, 0x7237e970, 0x0, 0x7237e964, 0x7237e958), at 0xfed315dc
[27] throw_and_post_jvmdi_exception(0x841810, 0x7237e9dc, 0x0, 0x0, 0x0, 0x0), at 0xfef4c8dc
[28] Runtime1::throw_null_exception(0x841810, 0xfac4c020, 0x0, 0x0, 0x0, 0x0), at 0xfef4cff8
[29] 0xfac4c050(0xe6f63eb8, 0x82ca99e0, 0x82c9aab0, 0x8296b198, 0xe6d561e0, 0xe6f63eb8), at 0xfac4c04f
[30] 0xfb402798(0xe6f55308, 0xe6f55308, 0x82ca99e0, 0x82c9aab0, 0xe6f55308, 0x764a7a00), at 0xfb402797
[31] 0xfb223fa8(0xe684c190, 0x82ca99e0, 0xe6f55290, 0xe6f55290, 0xe6f55290, 0xe6d561e0), at 0xfb223fa7
[32] 0xfb0a9924(0x829cef58, 0x82ca99e0, 0x82c9aab0, 0x3, 0xa, 0x764388c0), at 0xfb0a9923
[33] 0xfb0a924c(0x829cef58, 0x82ca99e0, 0x82c9aab0, 0x828fc8f0, 0x76438a78, 0x0), at 0xfb0a924b
[34] 0xfaf8ae20(0x829396b8, 0x829cef58, 0x82ca99e0, 0x82c9aab0, 0x82c69ea8, 0x0), at 0xfaf8ae1f
[35] 0xfaf00f10(0x829396b8, 0x82ca0568, 0x82ca99e0, 0x82c9aab0, 0x82c69ea8, 0x82c9aab0), at 0xfaf00f0f
[36] 0xfafc6794(0x76438a90, 0x82ca9418, 0x82c9aab0, 0x76438a20, 0x76438aa0, 0x829396b8), at 0xfafc6793
[37] 0xfb0a45fc(0x829c28d8, 0x82ca9418, 0x82c9aab0, 0xffffffff, 0x829c28d8, 0x76438a78), at 0xfb0a45fb
[38] 0xfaf8ae20(0x829396b8, 0x829c28d8, 0x82ca9418, 0x82c9aab0, 0x82c69ea8, 0x76438858), at 0xfaf8ae1f
[39] 0xfaf00f10(0x829396b8, 0x82ca0568, 0x82ca9418, 0x82c9aab0, 0x82940258, 0x82c9aab0), at 0xfaf00f0f
[40] 0xfafcd640(0x764385e0, 0x82ca0580, 0x82c9aab0, 0x3, 0x0, 0x82c9aab0), at 0xfafcd63f
[41] 0xfb4283c8(0xe684c190, 0x82ca0580, 0xe6d99878, 0x764385c8, 0x764385c8, 0x764385c8), at 0xfb4283c7
[42] 0xfac05da0(0x829cee78, 0x82ca0580, 0x82c9aab0, 0xfac15410, 0x764354c0, 0x7237f398), at 0xfac05d9f
[43] 0xfaf8ae20(0x829396b8, 0x829cee78, 0x82ca0580, 0x82c9aab0, 0x82c69ea8, 0x0), at 0xfaf8ae1f
[44] 0xfaf00f10(0x829396b8, 0x82ca0568, 0x82ca0580, 0x82c9aab0, 0x82c69ea8, 0x82c9aab0), at 0xfaf00f0f
[45] 0xfafc6794(0x76437f00, 0x82ca0380, 0x82c9aab0, 0x76435aa0, 0x76437f10, 0x829396b8), at 0xfafc6793
[46] 0xfb0ac0d4(0x82944450, 0x82ca0380, 0x82c9aab0, 0x764354c0, 0x82ca0380, 0x76435aa0), at 0xfb0ac0d3
[47] 0xfaf8ae20(0x829396b8, 0x82944450, 0x82ca0380, 0x82c9aab0, 0x82c69ea8, 0x0), at 0xfaf8ae1f
[48] 0xfaf00f10(0x829396b8, 0x82ca0568, 0x82ca0380, 0x82c9aab0, 0x764354c0, 0x82c9aab0), at 0xfaf00f0f
[49] 0xfafcd640(0x76435750, 0x82c9aa50, 0x82c9aab0, 0xffffffff, 0x0, 0x82c9aab0), at 0xfafcd63f
[50] 0xfb063340(0x829396b8, 0x82c9aa50, 0x82c98000, 0x764354c0, 0x764354c0, 0x764355e0), at 0xfb06333f
[51] 0xfaff55c0(0x828e9a18, 0x82c98000, 0x82c98000, 0x0, 0x82c98000, 0x2da), at 0xfaff55bf
[52] 0xfaff1f40(0x82c98000, 0xb, 0x1, 0xfac15410, 0x9, 0x7237fa00), at 0xfaff1f3f
[53] 0xfaf34074(0x829c27a8, 0x82c98000, 0x7ddb0330, 0xfac157a4, 0x82c69ea8, 0x828fe980), at 0xfaf34073
[54] 0xfaf1dd7c(0x7ddb0330, 0x7ddb03a0, 0x82c6c0c0, 0x7237fc0c, 0x0, 0x82c98000), at 0xfaf1dd7b
[55] 0xfac00438(0x7237fc50, 0x7237fe98, 0xa, 0xe69f1768, 0xfac0bc8c, 0x7237fde0), at 0xfac00437
[56] JavaCalls::call_helper(0x7237fe90, 0x7237fd18, 0x7237fdd8, 0x841810, 0x841810, 0x7237fd2c), at 0xfecfd9a4
[57] JavaCalls::call_virtual(0xfefe2000, 0x84be08, 0x7237fdcc, 0x7237fdc8, 0x7237fdd8, 0x841810), at 0xfed0f63c
[58] JavaCalls::call_virtual(0x7237fe90, 0x7237fe8c, 0x7237fe80, 0x7237fe78, 0x7237fe70, 0x841810), at 0xfed0f49c
[59] thread_entry(0x841810, 0x841810, 0x3bfb40, 0x84be08, 0x2d2ddc, 0xfed0f10c), at 0xfed0f424
[60] JavaThread::run(0x841810, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfed0f134
[61] _start(0x841810, 0x75da4e00, 0x0, 0x0, 0x0, 0x0), at 0xfecfc27c
(dbx)
JVM options used:
-Xms1800m -Xmx1800m -verbose:gc -XX:MaxPermSize=320m
-XX:+PrintTLAB -Xbatch (using default client VM)
The attached file hotspot_crash_1007.tar.gz contains the following information:
- core file
- dbx stack and threads
- output of showrev and uname
- exact/full hotspot error
- JVM options/flags used
(a 2nd attached file hotspot_crash_1006.tar.gz contains similar info from a
similar crash)
Currently, we are procuring additional info from as follows, just to see
of the crash is happening due to JNI OR due to JIT compiled code:
- to compile libcom_att_sas_utilities_XMLcrypto.so native library
on Sol8 prod server, along with -g option as well, to avoid
dbx demangle warnings
- procure info from the following test runs on 2 different prod servers:
1. -client, with -Xcheck:jni and -client and -XX:+PrintCompilation
2. -server, with -XX:-UseTLAB, -XX:+PrintGCDetails