JDK-6761744 : Hotspot crashes if process size limit is exceeded
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 5.0u16
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: solaris_10
  • CPU: x86
  • Submitted: 2008-10-21
  • Updated: 2013-09-12
  • Resolved: 2013-05-05
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7 Other
7u40Fixed hs24Fixed
Related Reports
Relates :  
Description
Solaris 10u5, x86:

When configuring too much heap, there should be a sensible error message
rather than crashes. This used to work in 1.4.2. With Java 5 and 6 I get crashes:


With 1.4.2 I get a useful error message:

$ /usr/jdk/java1.4.2/bin/java -version
java version "1.4.2_17"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_17-b06)
Java HotSpot(TM) Client VM (build 1.4.2_17-b06, mixed mode)
$ /usr/jdk/java1.4.2/bin/java -Xmx3072m -XX:MaxPermSize=1024m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.



With Java 5 I get a crash in the C2 compiler:

$ /usr/jdk/java5/bin/java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)
$ /usr/jdk/java5/bin/java -Xmx3072m -XX:MaxPermSize=1024m -version
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0xfe8ce9cb, pid=25116, tid=1
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_16-b02 mixed mode)
# Problematic frame:
# V  [libjvm.so+0xce9cb]
#
# An error report file with more information is saved as hs_err_pid25116.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#
Abort (core dumped)
$ dbx /usr/jdk/java5/bin/java core
HUP EMT FPE ALRM CLD WINCH TSTP CONT WAITING LWP CANCEL 
dbx: Value of `$DBX_disassembler_version' must be one of `autodetect', `x86_32', or `x86_64'
Use the `help' command for more information.
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.6' in your .dbxrc
Reading java
dbx: internal warning: writable memory segment 0xfe5a0000[32768] of size 0 in core
core file header read successfully
Reading ld.so.1
Reading libthread.so.1
Reading libdl.so.1
Reading libc.so.1
Reading libjvm.so
Reading libsocket.so.1
Reading libsched.so.1
Reading libCrun.so.1
Reading libm.so.1
Reading libnsl.so.1
Reading libm.so.2
Reading libscf.so.1
Reading libdoor.so.1
Reading libuutil.so.1
Reading libgen.so.1
dbx: core file read error: address 0xfe648000 not available
t@null (l@1) terminated by signal ABRT (Abort)
0xfef44aa7: __lwp_kill+0x0007:  jae      __lwp_kill+0x15        [ 0xfef44ab5, .+0xe ]
dbx> where
=>[1] __lwp_kill(0x1, 0x6), at 0xfef44aa7 
  [2] _thr_kill(0x1, 0x6), at 0xfef42250 
  [3] raise(0x6), at 0xfeef1217 
  [4] abort(0xfedfa000, 0x8045530, 0xfed24a9a, 0x1, 0x8073f50, 0x0), at 0xfeed1629 
  [5] os::abort(0x1), at 0xfecc823f 
  [6] VMError::report_and_die(0x8045548, 0x8045740, 0xfef62a00, 0xfedfa000, 0xb, 0xfedc67d2), at 0xfed24a9a 
  [7] JVM_handle_solaris_signal(0xb, 0x804598c, 0x804578c, 0x1), at 0xfe975158 
  [8] signalHandler(0xb, 0x804598c, 0x804578c), at 0xfe9748d0 
  [9] __sighndlr(0xb, 0x804598c, 0x804578c, 0xfe9748b0), at 0xfef43e6f 
  [10] call_user_handler(0xb, 0x804598c, 0x804578c), at 0xfef3a30e 
  [11] sigacthandler(0xb, 0x804598c, 0x804578c, 0xf, 0x0, 0x4), at 0xfef3a48e 
  ---- called from signal handler with signal 11 (SIGSEGV) ------
  [12] instanceKlass::find_method(0x0, 0xf7e00ce0, 0xf7e00ce0), at 0xfe8ce9cb 
  [13] instanceKlass::find_method(0xf7e00ce8, 0xf7e00ce0, 0xf7e00ce0), at 0xfe8cedd2 
  [14] ciInstanceKlass::find_method(0x8116788, 0x8116788, 0x8116788), at 0xfe950607 
  [15] Compile::register_library_intrinsics(0x8046098), at 0xfe9a2a5f 
  [16] Compile::Init(0x8046098, 0x0), at 0xfe99d29a 
  [17] Compile::Compile(0x8046098, 0x80464e0, 0xfea35450, 0xfea3e380, 0xfedb65fd, 0x1, 0x1, 0x1, 0x0), at 0xfea16a94 
  [18] OptoRuntime::generate_stub(0x80464e0, 0xfea35450, 0xfea3e380, 0xfedb65fd, 0x1, 0x1, 0x1, 0x0), at 0xfea1736a 
  [19] OptoRuntime::generate(0x80464e0), at 0xfea864ed 
  [20] compiler2_init(0xfedfa000, 0x8046680, 0xfea58e48, 0x806f60c, 0xfee4f494, 0xfedfa000), at 0xfea7f584 
  [21] init_globals(0x806f60c, 0xfee4f494, 0xfedfa000, 0x8046590, 0xfefd069a, 0x8046559), at 0xfea6cf95 
  [22] Threads::create_vm(0x80466c8, 0x80466a4), at 0xfea58e48 
  [23] JNI_CreateJavaVM(0x8046efc, 0x8046f00, 0x80466c8), at 0xfea731df 
  [24] main(0x3, 0x806f67c, 0x8046f54), at 0x80518f8 
dbx> 



With Java6 I get a crash in _memset:
$ /usr/jdk/java6/bin/java -version
java version "1.6.0_10"
Java(TM) SE Runtime Environment (build 1.6.0_10-b33)
Java HotSpot(TM) Server VM (build 11.0-b15, mixed mode)
$ /usr/jdk/java6/bin/java -Xmx3072m -XX:MaxPermSize=1024m -version
#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xfeea5320, pid=25123, tid=2
#
# Java VM: Java HotSpot(TM) Server VM (11.0-b15 mixed mode solaris-x86)
# Problematic frame:
# C  [libc.so.1+0x25320]  _memset+0xc0
#
# An error report file with more information is saved as:
# /home/tv9055/hs_err_pid25123.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#
Abort (core dumped)
$

Comments
The actual fix is in duplicated CR's (JDK-8013863 and JDK-8014599, automatically created somehow when merging) and it has a jtreg test.
17-05-2013

Had wrong fix version, so a backport issue was created JDK-8013863.
05-05-2013

EVALUATION Need to check the total heap size (young gen + old gen + perm gen) in ParallelScavengeHeap::initialize().
23-07-2009