Bug ID: JDK-6644629 JVM dies with little/misleading problem information during excessive creation of ORB threads

Type: Bug
Component: hotspot
Sub-Component: runtime
Affected Version: 5.0u13

Priority: P2
Status: Closed
Resolution: Duplicate
OS: solaris_10
CPU: sparc

Submitted: 2007-12-21
Updated: 2010-08-18
Resolved: 2008-09-23

Other
5.0u14Resolved

If thousands of threads are created for ORB connections (using the unrecommended com.sun.jndi.cosnaming.CNCtx initial context) the JVM frequently dies with no, little or misleading information about the cause. 
With Java 5 there is no hint about ressource consumption. Java 6 throws an OutOfMemoryError although Java heap is not exhausted and virtual process size is less than 1 GB. The JVM in most cases crashes with SIGBUS or SIGSEGV but without either a core or a hs_err_* file. (The problem is even more hidden if the JVM is embedded into S1AS 8.2)

WORK AROUND A similar problem still exists in Java 1.6.0_16 on Linux: If with a bad Apache HTTPD2/Tomcat configuration thousands of sockets are opened, the Tomcat JVM finally dies disgracefully with no traces (hs_err_pid, core file or similar) left behind.
30-09-2009
WORK AROUND A similar problem still exists in Java 1.6.0_16 on Linux: If with a bad Apache HTTPD2/Tomcat configuration thousands of sockets are opened, the Tomcat JVM finally dies disgracefully with no traces (hs_err_pid, core file or similar) left behind.
30-09-2009
EVALUATION Thank you so much for your response. I would like to close this as a duplicate of 6302804. Hopefully we can address the comprehensive solution to our C heap memory usage in the next major release. I'm not sure how that affects the escalation reported against this bug. Thanks.
23-09-2008
EVALUATION Yes, RFE 6694099 resolves the difficulty of debugging in scenarios of vm_exit_out_of_memory. Without, we were not able to obtain a core file, which has made it very difficult to get to the root cause in my particular customer scenario. Now, with that RFE in place, we are able to get a core file and thus being in a better position to look at the problem.
20-09-2008
EVALUATION When the VM runs out of C heap, which happens during thread creation, there isn't a good recovery. Some of the VM operations simply cause a crash like the compiler memory allocation for it's only IL, for instance. This is a known long term work item in the Hotspot VM, because there isn't a "fix". There's some redesigning involved. This has been traditionally been tracked as bug 6302804, which we've been considering this as a duplicate. There is also a fix requested by several customers that we dump core when these unhandled errors occur in the VM so that the customer can do analysis on the core file and know how to resolve the problem, if it's native or C heap exhaustion. This bug is 6694099. Please let us know if the latter bug resolves the difficulty of debugging reported in this problem report.
19-09-2008
EVALUATION Reducing JavaHeap and thread stack space enabled generation of core. Refer Workaround section for more details *** (#1 of 1): [ UNSAVED ] ###@###.###
01-04-2008
WORK AROUND Reducing JavaHeap and thread stack space enabled generation of core. Raising an RFE, to provide core dump for such scenarios, is being considered Below is the detailed anaylsis ------------------------------------ The application server dumped core for the first time when -XX:PrintCompilation flag was set. This flag does not have anything to do with core generation. All it does is, prints out the compilation steps followed by the JVM during the process execution. Since this alteration had more memory requriements, the process was at a different execution point (compared to previous iterations) when out of memory was hit. Since there was no specific handler for this, default handler was invoked and hence the core dump. To confirm this, we reduced the Java Heap space and the thread stack space (using the options java -Xmx1024m -Xms1024m -Xss128k -- Note Xmx specifies max Java heap size and Xms specifies the min Java Heap size. Xss specifies the thread stack space for each thread). Reducing the Java heap space and thread stack space makes more room for the virtual address space and the native C heap. With this situation, when the core dumped, it revealed the following. Below is the core dump. current thread: t@11 =>[1] __lwp_kill(0x0, 0x6, 0x0, 0x6, 0xfc00, 0x0), at 0xfef469ec [2] raise(0x6, 0x0, 0xfef72db8, 0xfef298e4, 0xffffffff, 0x6), at 0xfeee4a04 [3] abort(0xf997e2a8, 0x1, 0x0, 0xad318, 0xfef71318, 0x0), at 0xfeec1118 [4] os::abort(0xfed48d84, 0x2cf48, 0xfed6d7c4, 0xf997e2a8, 0x1, 0xfe9d2bb6), at 0xfde907cc [5] vm_exit_out_of_memory(0xfed82ce8, 0x2e400, 0x44590, 0x1e5750, 0x4, 0xfe5be32f), at 0xfd991d9c [6] Chunk::operator new(0x0, 0x7fec, 0x26a8, 0x44a18, 0xfe1bac23, 0xfed18d0c), at 0xfd62f85c [7] Arena::grow(0x1e5a70, 0x2bf0, 0x1e5a70, 0x7e8cc0, 0x1e5a70, 0xfecb98dc), at 0xfd6313e4 [8] GenericGrowableArray::GenericGrowableArray(0xf997e554, 0xafc, 0x0, 0xddf4, 0xfecb98dc, 0xfe4df0da), at 0xfd8efedc [9] PhaseIdealLoop::build_loop_tree(0xf997eae8, 0x0, 0xffffffff, 0xfffffff8, 0x0, 0xf997e554), at 0xfdda4660 [10] PhaseIdealLoop::PhaseIdealLoop(0xf997eae8, 0x1e5750, 0x0, 0x1, 0x1ccfafb0, 0x1600), at 0xfdda1f80 [11] Compile::Optimize(0xf997f2a0, 0xfed48d58, 0xfe3e0a8e, 0xfecb98dc, 0xfe3dde88, 0x2b), at 0xfd7bb3f8 [12] Compile::Compile(0x0, 0xfe3dace4, 0x1df9d0, 0xfed1d790, 0x1e5a70, 0xf997fbe4), at 0xfd7b6260 [13] C2Compiler::compile_method(0x1df9d0, 0xf997fbe4, 0x1ce26348, 0xffffffff, 0x1, 0xf997f358), at 0xfd6b4214 [14] CompileBroker::invoke_compiler_on_method(0x13b8190, 0x1e5750, 0xfed39ea0, 0x1f44, 0x1ce26348, 0x0), at 0xfd7c9560 [15] CompileBroker::compiler_thread_loop(0xfed6c2d8, 0x1e5750, 0x13b8190, 0xfefefefe, 0xfed39fc4, 0xfed2815c), at 0xfd7c7 [16] JavaThread::thread_main_inner(0x1e5750, 0x0, 0xfed486f8, 0xfecb98dc, 0x2f0c0, 0x1e5750), at 0xfe04ae50 [17] _start(0x1e5750, 0x2d348, 0xb, 0xfecb98dc, 0x2d000, 0xfe04a8a8), at 0xfde8eae0 _Analysis of the core._ At frame 7, there is an attempt to allocate more memory using new /hotspot/src/share/vm/adlc/arena.cpp void* Arena::grow( size_t x ) { ..... register Chunk k = _chunk; // Get filled-up chunk address _chunk = new (len) Chunk(len); <---------------------- This "new" errors since there is no more space .......... return result; } /To further confirm, here is the dbx output (dbx) frame 7 0xfd6313e4: grow+0x003c: call operator new ! 0xfd62f714 (dbx) frame 6 0xfd62f85c: operator new+0x0148: ld [%fp + 72], %f31 / void vm_exit_out_of_memory(size_t size, const char name) { vm_abort(); } void vm_abort() { ...... os::abort(PRODUCT_ONLY(false)); <--passed false for dumping core. Since we have a fast debug version this time, it dumped core ........ } void os::abort(bool dump_core) { os::shutdown(); if (dump_core) { #ifndef PRODUCT "Dumping core" #endif ::abort(); // dump core (for debugging) } ::exit(1); } Note: This is the core generated with the fast debug version. However bringing down the java heap and thread stack space gives a similar core dump. It also shows the reason why core was not dumped in the previous iterations. That is because, JVM is shipped in two versions. The fast debug version and the product version. The product version is what the enterprises use for routine transactions and fast debug is provided to enable debugging in case of exceptions which require low level tracing. For this particular situation, since the machine is already out of memory, the JVM is designed not to dump core but abort with a message "OutOfMemory". (The ifndef clause below shows this check). However when using the fast debug version, the core is dumped. -- Regards Sunita Koppar Java Sustaining Engineer DID: 91-80-69927738
01-04-2008

Duplicate :	JDK-6302804 - Hotspot VM dies ungraceful death when C heap is exhausted in various places.
Relates :	JDK-4343963 - hotspot dumps core on minor vm exits (e.g. mistyped library name)
Relates :	JDK-6694099 - Hotspot vm_exit_out_of_memory should dump core
Relates :	JDK-6362318 - (thread) uncaught exception handler can be silenced by OutOfMemoryError