There are currently 3 opened cases from 3 different customers seeing this problem with
weblogic:
case# 65220105 (5.0u4/sol 9),
case# 65219952 (5.0u5/sol 8),
case# 65189765 (5.0u8/sol 10).
All showing the same stack trace.
Data locations (core, hs_err etc) :
65219952: /net/cores.central/cores/65219952/Pkg_App_Data
65189765: /net/cores.central/cores/dir31/65189765/oct_26/core4
65220105: /net/cores.central/cores/65220105
Here's a detail investigation on case 65219952:
dbx:
----
current thread: t@42
=>[1] libthread.so.1:__sigprocmask(0x0, 0xc7200618, 0x0, 0x0, 0x0, 0x0), at 0xff3692f0
[2] libthread.so.1:_resetsig(0xff36b8f4, 0x0, 0x0, 0xc7201d98, 0xff37c000, 0x0), at 0xff35e594
[3] libthread.so.1:_sigon(0xc7201d98, 0xff3838b0, 0x6, 0xc72006ec, 0xc7201d98, 0xff383224), at 0xff35dd34
[4] libthread.so.1:_thrp_kill(0x0, 0x2a, 0x6, 0xff37c000, 0x2a, 0xff2c0458), at 0xff360d90
[5] libc.so.1:raise(0x6, 0x0, 0x0, 0xffffffff, 0xff2c03c4, 0x8000), at 0xff24bdd8
[6] libc.so.1:abort(0xff2bc008, 0xfe39c000, 0xfe259178, 0xfe3f07ec, 0x0, 0x34450), at 0xff235984
[7] libjvm.so:os::abort(0x1, 0x0, 0xfe3cff3c, 0xfe39c000, 0x7134, 0x7000), at
0xfe253234
[8] libjvm.so:VMError::report_and_die(0x0, 0xfe3f6b78, 0xfe3f07c4, 0x1, 0xfe2572ac, 0xfe3f07c4), at 0xfe2d2dc0
[9] libjvm.so:JVM_handle_solaris_signal(0xb, 0xc7200e88, 0xc7200bd0, 0x7400, 0xfe3ef7f4, 0xd4e4b0), at 0xfde73d44
[10] libthread.so.1:__sighndlr(0xb, 0xc7200e88, 0xc7200bd0, 0xfde73260, 0xc7201e40, 0xc7201e30), at 0xff36b218
---- called from signal handler with signal 11 (SIGSEGV) ------
[11] libjvm.so:InterpreterRuntime::exception_handler_for_exception(0x51c9f4, 0x0, 0xd4e4b0, 0x7000, 0x51ca00, 0xd40590), at 0xfddc7b34
[12] 0xf8409ff8(0xee473d40, 0xb6, 0x0, 0xf8415cf4, 0x2ef4, 0xc7201260), at 0xf8409ff8
[13] 0xf84058fc(0xe1e23e40, 0xb6, 0x0, 0xf8415cf4, 0x5d4c, 0xc72012f0), at 0xf84058fc
[14] 0xf8405764(0x0, 0xdf421fb8, 0x0, 0xf8415d10, 0xdf421fb8, 0xc7201390), at
0xf8405764
[15] 0xf844dff4(0xe1e23d40, 0xe1e23e40, 0xe1e23ee0, 0xff130000, 0xdf421fb8, 0xd8aa6468), at 0xf844dff4
[16] 0xf9020c48(0xd4aa98a8, 0xd88f3928, 0xd029fe80, 0xf8415008, 0xd0297408, 0xd88f80a0), at 0xf9020c48
[17] 0xf8465bf8(0xe1e23ee0, 0xb6, 0x0, 0xf84147c8, 0x11d03, 0xc7201460), at 0xf8465bf8
SA's pstackproc:
---------------
0xff3692f0 __sigprocmask + 0x8
0xff35dd34 _sigon + 0xd0
0xff360d90 _thrp_kill + 0xf8
0xff24bdd8 raise + 0x40
0xff235984 abort + 0x100
0xfe253234 void os::abort(int) + 0x58
0xfe2d2dc0 void VMError::report_and_die() + 0xc84
0xfde73d44 JVM_handle_solaris_signal + 0xaac
0xff36b218 __sighndlr + 0xc
0xff36821c sigacthandler + 0x708
0xfddc7b34 unsigned char*InterpreterRuntime::exception_handler_for_exception(JavaThread*,oopDesc*) + 0x3c4
0xf8409ff8 * weblogic.servlet.internal.ServletInputStreamImpl.skip(long) bci:5
line:191 methodOop:0xd06f5160 (Interpreted frame)
0xf84058fc * weblogic.servlet.internal.ServletRequestImpl.reset() bci:42 line:147 methodOop:0xd06ec128 (Interpreted frame)
0xf8405764 * weblogic.servlet.internal.MuxableSocketHTTP.requeue() bci:18 line:177 methodOop:0xd3127408 (Interpreted frame)
0xf844dff4 <C2IAdapter>
0xf9020c48 0xf9020c48 * weblogic.servlet.internal.VirtualConnection.requeue()
bci:4 line:323 methodOop:0xd3138dc8 (Compiled frame)
0xf8465bf8 <I2CAdapter>
0xf8405764 * weblogic.servlet.internal.ServletRequestImpl.run() bci:269 line:1319 methodOop:0xd06f09d0 (Interpreted frame)
0xf8405c2c * weblogic.work.ExecuteThread.execute(java.lang.Runnable) bci:29 line:207 methodOop:0xd115a080 (Interpreted frame)
0xf8454c38 <C2IAdapter>
0xf8d04460 0xf8d04460 * weblogic.work.ExecuteThread.run() bci:42 line:179 methodOop:0xd1159eb8 (Compiled frame)
0xf847fa78 <OSRAdapter>
0xf8400218 <StubRoutines>
0xfdd97ad8 void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*) + 0x5a0
0xfdebfc54 void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,symbolHandle,symbolHandle,Thread*) + 0x188
0xfdedec98 void thread_entry(JavaThread*,Thread*) + 0x134
0xfdeda834 void JavaThread::run() + 0x1d8
0xfe252d50 void*_start(void*) + 0x208
0xff36b11c _thread_start + 0x40
SA's jstackproc:
---------------
Thread t@42: (state = IN_VM, current Java SP = 0xc72011d8
)
- weblogic.servlet.internal.ServletInputStreamImpl.skip(long) @bci=5, line=191, pc=0xf840a000, methodOop=0xd06f5160 (Interpreted frame)
- weblogic.servlet.internal.ServletRequestImpl.reset() @bci=42, line=147, pc=0xf8405904, methodOop=0xd06ec128 (Interpreted frame)
- weblogic.servlet.internal.ServletRequestImpl.reset() @bci=42, line=147, pc=0xf8405904, methodOop=0xd06ec128 (Interpreted frame)
- weblogic.servlet.internal.MuxableSocketHTTP.requeue() @bci=18, line=177, pc=0xf840576c, methodOop=0xd3127408 (Interpreted frame)
- weblogic.servlet.internal.VirtualConnection.requeue() @bci=4, line=323, pc=0xf9020c50, methodOop=0xd3138dc8 (Compiled frame)
- weblogic.servlet.internal.ServletRequestImpl.run() @bci=269, line=1319, pc=0xf840576c, methodOop=0xd06f09d0 (Interpreted frame)
- weblogic.work.ExecuteThread.execute(java.lang.Runnable) @bci=29, line=207, pc=0xf8405c34, methodOop=0xd115a080 (Interpreted frame)
- weblogic.work.ExecuteThread.run() @bci=42, line=179, pc=0xf8d04468, methodOop=0xd1159eb8 (Compiled frame)
ServletInputStreamImpl.jad
--------------------------
public long skip(long l)
throws IOException
{
return in.skip(l);
// 0 0:aload_0
// 1 1:getfield #2 <Field InputStream in>
// 2 4:lload_1
---> crash here
// 3 5:invokevirtual #14 <Method long InputStream.skip(long)>
// 4 8:lreturn
}
JVM options:
----------
dbx> mainargs
main's fp = 0xffbee5f8
argc = 24
argv = 0xffbee65c
envp = 0xffbee6c0
Arguments:
0xffbee914: "/export/webapps/scommon/jdk/jdk1.5.0_05/bin/java"
0xffbee945: "-server"
0xffbee94d: "-Xms512m"
0xffbee956: "-Xmx512m"
0xffbee95f: "-Dsun.net.inetaddr.ttl=60"
0xffbee979: "-Dsun.net.inetaddr.ttl=60"
0xffbee993: "-XX:MaxPermSize=128m"
0xffbee9a8: "-DlogAnalyzer=true"
0xffbee9bb: "-Dfanniemae.app.name=clm"
0xffbee9d4: "-Dweblogic.log.RedirectStdoutToServerLogEnabled=true"
0xffbeea09: "-Dweblogic.alternateTypesDirectory=/export/webapps/scommon/WARP/WL9/4.2.0.2/mbeantypes"
0xffbeea60: "-DWarpLogLevel=INFO"
0xffbeea74: "-Dweblogic.alternateTypesDirectory=/export/webapps/scommon/WARP/WL9/4.2.0.2/mbeantypes"
0xffbeeacb: "-DWarpLogLevel=INFO"
0xffbeeadf: "-classpath"
0xffbeeaea: "/export/appl/website/CLM-EXT/applications/Resources/ojdbc14.jar:/export/webapps/sbea/weblogic910/host_patches/CR258450_910.jar:/export/webapps/scommon/jdk/jdk1.5.0_05/lib/tools.jar:/export/webapps/sbea/weblogic910/server:/export/webapps/sbea/weblogic910/server/lib/weblogic.jar:/export/webapps/sbea/weblogic910/server/lib/xbean.jar::/export/webapps/scommon/WARP/WL9/4.2.0.2/mbeanlib/concurrent.jar:/export/webapps/scommon/WARP/WL9/4.2.0.2/mbeanlib/WARPSecurityProvidersStartup.jar:/export/webapps/scommon/WARP/WL9/4.2.0.2/mbeanlib/RAM_APIClient.jar::/export/webapps/scommon/WARP/WL9/4.2.0.2/mbeanlib/concurrent.jar:/export/webapps/scommon/WARP/WL9/4.2.0.2/mbeanlib/WARPSecurityProvidersStartup.jar:/export/webapps/scommon/WARP/WL9/4.2.0.2/mbeanlib/RAM_APIClient.jar:"
0xffbeede9: "-Dweblogic.Name=CLM-EXT-ap01"
0xffbeee06: "-Dbea.home=/export/webapps/sbea"
0xffbeee26: "-Dweblogic.management.server=t3://pcust-admin01a:1861"
0xffbeee5c: "-Dweblogic.ProductionModeEnabled=true"
0xffbeee82: "-Dweblogic.RootDirectory=/export/appl/website/weblogic/CLM-EXT"0xffbeeec1: "-Dweblogic.system.BootIdentityFile=/export/appl/website/weblogic/CLM-EXT/security/boot.properties"
0xffbeef23: "-Djava.security.policy=/export/webapps/sbea/weblogic910/server/lib/weblogic.policy"
0xffbeef76: "weblogic.Server"