JDK-4489387 : -Xrunhprof crashed jvm or got program run to hang
  • Type: Bug
  • Component: vm-legacy
  • Sub-Component: jvmpi
  • Affected Version: 1.3.1_07,1.4.0
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: generic,solaris_7,solaris_8
  • CPU: generic,sparc
  • Submitted: 2001-08-07
  • Updated: 2013-11-01
  • Resolved: 2001-12-04
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
1.4.0 beta3Fixed
Related Reports
Duplicate :  
JDK-4796505 - HotSpot hangs during dispatch of JVMPI_EVENT_GC_START
Relates :  
JDK-4478223 - "oopmap not found" assertion failure in jvmpi heap dump collection
Relates :  
JDK-4496749 - C2: missing METHOD_EXIT events
Relates :  
JDK-4507533 - HPROF: No heap dump generated (java.exe crashes or HPROF ERROR: heap dump size < 0)
Relates :  
JDK-4636055 - HPROF: JVM crashes during Shutdown when cpu=times with many threads, finalizatio
Relates :  
JDK-4835665 - HPROF: -Xrunprof:heap=dump causes OutOfMemoryError
Relates :  
JDK-4491764 - JVMPI method entry, exit events cause client VM to crash with Unexpected Signal.
Relates :  
JDK-4507553 - VM hangs on Linux when running hprof
Description
I ran -Xrunhprof  options:
  -Xrunhprof:heap=dump
  -Xrunhprof:heap=all
  -Xrunhprof:cpu=times
  -Xrunhprof:cpu=samples

with a few benchmark programs(totally 3 programs) against jdk1.4 b74, only  -Xrunhprof:cpu=samples worked ok for all programs, the other three either 
crashed jvm or got the run to hang(and ignored cltr-C signal).

To reproduce:

% uname -a
SunOS vispro 5.7 Generic_106541-16 sun4u sparc SUNW,Ultra-30

% jdk1.4/bin/java -version
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0-beta_refresh-b74)
Java HotSpot(TM) Client VM (build 1.4.0-beta_refresh-b74, mixed mode)

% cd /home/zzhang/bugs/footprint
% jdk1.4/bin/java -Xrunhprof:cpu=times -cp jmol-0.2/jars/jmol.jar -Djmol.home=jmol-0.2 jmol.Jmol
HPROF ERROR: thread local table NULL in method exit a00dc
HPROF ERROR: thread local table NULL in method exit a00dc
HPROF ERROR: thread local table NULL in method exit a0a8c
HPROF ERROR : stack underflow in method exit
HPROF ERROR : stack underflow in method exit

Unexpected Signal : 11 occurred at PC=0xFE59C8A4
Function=JVM_IsInterrupted+0xC30
Library=/net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/client/libjvm.so


Dynamic libraries:
0x10000         /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4/bin/java
0xff360000      /usr/lib/libthread.so.1
0xff3a0000      /usr/lib/libdl.so.1
0xff280000      /usr/lib/libc.so.1
0xff260000      /usr/platform/SUNW,Ultra-30/lib/libc_psr.so.1
0xfe400000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/client/libjvm.so
0xff210000      /usr/lib/libCrun.so.1
0xff1f0000      /usr/lib/libsocket.so.1
0xff100000      /usr/lib/libnsl.so.1
0xff1c0000      /usr/lib/libm.so.1
0xff240000      /usr/lib/libw.so.1
0xff0e0000      /usr/lib/libmp.so.2
0xff090000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/native_threads/libhpi.so
0xff060000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/libverify.so
0xff030000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/libjava.so
0xfe7e0000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/libzip.so
0xfe200000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/libhprof.so
0xf3f00000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/libawt.so
0xfa480000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/libmlib_image.so
0xfc920000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/motif21/libmawt.so
0xf3c80000      /usr/dt/lib/libXm.so.4
0xfc810000      /usr/openwin/lib/libXt.so.4
0xfa7d0000      /usr/openwin/lib/libXext.so.0
0xfe150000      /usr/openwin/lib/libXtst.so.1
0xf3b80000      /usr/openwin/lib/libX11.so.4
0xfa6a0000      /usr/openwin/lib/libdps.so.5
0xfa7b0000      /usr/openwin/lib/libSM.so.6
0xfa5d0000      /usr/openwin/lib/libICE.so.6
0xf3a80000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/libfontmanager.so
0xfa790000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/libnio.so
0xfa5b0000      /usr/lib/libposix4.so.1
0xfa450000      /net/vispro/export/home2/zzhang/sugarland/java/jdk1.4_b74/jre/lib/sparc/libnet.so
0xfa590000      /usr/lib/libaio.so.1

Local Time = Tue Aug  7 16:19:05 2001
Elapsed Time = 10
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002D7 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Client VM (1.4.0-beta_refresh-b74 mixed mode)
#
# An error report file has been saved as hs_err_pid9941.log.
# Please refer to the file for further information.
#
Abort


% jdk1.4/bin/java -Xrunhprof:heap=all -cp jmol-0.2/jars/jmol.jar -Djmol.home=jmol-0.2 jmol.Jmol

the run would hang there.

to run the other two programs, under the same directory:

% jdk1.4/bin/java  -cp TeaTimeJ/teatime.jar mm.project.teatime

% cd mailpuccino
% jdk1.4/bin/java -cp mailpuccino.jar Mailpuccino


jdk1.3.1 seemed to work file with all these.

Name: rmT116609			Date: 09/14/2001


java version "1.4.0-beta2"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0-beta2-b77)
Java HotSpot(TM) Client VM (build 1.4.0-beta2-b77, mixed mode)


-Xrunhprof:cpu=times dumps core in jdk1.4beta2.

-Xrunhprof:cpu=old dumps core in jdk1.4beta2.

-Xrunhprof:monitor=y sometimes dumps core (more often than in
 jdk1.2 and jdk1.3 (I wonder what happened to that bug report)).

-Xrunhprof:cpu=samples seems to work.

% cat hello.java
public class hello
{
    public static void main(String args[])
    {
        System.out.println("hello world");
    }
}
% /usr/java1.4/bin/javac hello.java
% /usr/java1.4/bin/java hello
hello world
% /usr/java1.4/bin/java -Xrunhprof:cpu=times hello
HPROF ERROR: thread local table NULL in method exit 9ef44
HPROF ERROR: thread local table NULL in method exit 9ef44
HPROF ERROR: thread local table NULL in method exit 9f8f4
HPROF ERROR : stack underflow in method exit
HPROF ERROR : stack underflow in method exit

Unexpected Signal : 11 occurred at PC=0xFE550710
Function=JVM_IsSupportedJNIVersion+0x30B30
Library=/usr/java1.4/jre/lib/sparc/client/libjvm.so

Current Java thread:

Dynamic libraries:
0x10000         /usr/java1.4/bin/java
0xff360000      /usr/lib/libthread.so.1
0xff3a0000      /usr/lib/libdl.so.1
0xff200000      /usr/lib/libc.so.1
0xff330000      /usr/platform/SUNW,UltraSPARC-IIi-cEngine/lib/libc_psr.so.1
0xfe400000      /usr/java1.4/jre/lib/sparc/client/libjvm.so
0xff2e0000      /usr/lib/libCrun.so.1
0xff1e0000      /usr/lib/libsocket.so.1
0xff100000      /usr/lib/libnsl.so.1
0xff0d0000      /usr/lib/libm.so.1
0xff310000      /usr/lib/libw.so.1
0xff0b0000      /usr/lib/libmp.so.2
0xff080000      /usr/java1.4/jre/lib/sparc/native_threads/libhpi.so
0xff050000      /usr/java1.4/jre/lib/sparc/libverify.so
0xff020000      /usr/java1.4/jre/lib/sparc/libjava.so
0xfe7e0000      /usr/java1.4/jre/lib/sparc/libzip.so
0xfe1d0000      /usr/java1.4/jre/lib/sparc/libhprof.so

Local Time = Fri Sep 14 11:31:32 2001
Elapsed Time = 1
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002D7 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Client VM (1.4.0-beta2-b77 mixed mode)
#
# An error report file has been saved as hs_err_pid8575.log.
# Please refer to the file for further information.
#
Abort (core dumped)
(134)
% /usr/java1.4/bin/java -Xrunhprof:cpu=old hello
HPROF ERROR: thread local table NULL in method exit 9ef54
HPROF ERROR: thread local table NULL in method exit 9ef54
HPROF ERROR: thread local table NULL in method exit 9f904
HPROF ERROR : stack underflow in method exit
HPROF ERROR : stack underflow in method exit

Unexpected Signal : 11 occurred at PC=0xFE550710
Function=JVM_IsSupportedJNIVersion+0x30B30
Library=/usr/java1.4/jre/lib/sparc/client/libjvm.so

Current Java thread:

Dynamic libraries:
0x10000         /usr/java1.4/bin/java
0xff360000      /usr/lib/libthread.so.1
0xff3a0000      /usr/lib/libdl.so.1
0xff200000      /usr/lib/libc.so.1
0xff330000      /usr/platform/SUNW,UltraSPARC-IIi-cEngine/lib/libc_psr.so.1
0xfe400000      /usr/java1.4/jre/lib/sparc/client/libjvm.so
0xff2e0000      /usr/lib/libCrun.so.1
0xff1e0000      /usr/lib/libsocket.so.1
0xff100000      /usr/lib/libnsl.so.1
0xff0d0000      /usr/lib/libm.so.1
0xff310000      /usr/lib/libw.so.1
0xff0b0000      /usr/lib/libmp.so.2
0xff080000      /usr/java1.4/jre/lib/sparc/native_threads/libhpi.so
0xff050000      /usr/java1.4/jre/lib/sparc/libverify.so
0xff020000      /usr/java1.4/jre/lib/sparc/libjava.so
0xfe7e0000      /usr/java1.4/jre/lib/sparc/libzip.so
0xfe1d0000      /usr/java1.4/jre/lib/sparc/libhprof.so

Local Time = Fri Sep 14 11:31:36 2001
Elapsed Time = 1
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002D7 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Client VM (1.4.0-beta2-b77 mixed mode)
#
# An error report file has been saved as hs_err_pid8578.log.
# Please refer to the file for further information.
#
Abort (core dumped)
(134)
% /usr/java1.4/bin/java -Xrunhprof:monitor=y hello
hello world
Dumping contended monitor usage ... done.
% /usr/java1.4/bin/java -Xrunhprof:monitor=y hello
hello world
Dumping contended monitor usage ... done.
% /usr/java1.4/bin/java -Xrunhprof:monitor=y hello
hello world
Dumping contended monitor usage ... done.
% /usr/java1.4/bin/java -Xrunhprof:monitor=y hello
Segmentation fault (core dumped)
(139)
%
(Review ID: 131918)
======================================================================
Comments
CONVERTED DATA BugTraq+ Release Management Values COMMIT TO FIX: merlin-beta3 FIXED IN: merlin-beta3 INTEGRATED IN: merlin-beta3 VERIFIED IN: merlin-rc1
14-06-2004
EVALUATION ###@###.### 2001-08-15 I can reproduce the crash when running hprof cpu=times option on all 3 applications (Jmol, TeaTime, and Mailpuccino). The following problems are found (see comments for details): 1. "oopmap not found" assertion using JDK1.4 build 74 See 4478223. jvmpi_method_exit is a JRT_ENTRY but no oopmap info is generated. 2. segmentation fault in the build 75 client VM when compiling Java method See 4492473. It's a regression introduced in build 75. 3. various hprof errors: HPROF ERROR: thread local table NULL in method exit 11a1e0 HPROF ERROR: thread local table NULL in method exit 11a1e0 HPROF ERROR: thread local table NULL in method exit 11a690 HPROF ERROR : stack underflow in method exit HPROF ERROR : stack underflow in method exit default to Motif 2.1, os is: 5.8 and tons of this error: HPROF ERROR: method on stack top != method exiting.. Other three hprof options (cpu=samples, heap=dump, heap=all) work fine. I am no longer able to reproduce the hang using JDK1.4 b74 and b75. ###@###.### 2001-09-24 Dave Cox has a fix for 4478223. I have ran java -Xrunhprof:cpu=times on the following applications on Solaris Sparc and Solaris x86: Jmol TeaTime mailpuccino SwingSet2 I found and fixed two other problems: 1. VM hang due to ObjectMonitor::raw_enter() blindly cast a thread pointer to JavaThread*. It made a bad assumption that it must be called only by JavaThread. VMThread can also acquire a raw monitor when it posts a GC_START event to the agent, which enters a raw monitor for data synchronization. So fix the hotspot VM to check for the thread's type before casting. 2. Deadlock on a raw monitor. One thread is processing THREAD_END event and enter a raw monitor. While holding the lock, it makes JVMPI call (GetThreadLocalStorage) which gives a chance to the VM to block the thread for other operation such as GC. If the thread is blocked and start GC, VM Thread will post a GC_START event to the agent, which also acquires the raw monitor first before further processing. Fix hprof to make sure JVMPI calls are called while either not holding a lock or GC is disabled for that event. Dave Cox will putback the fix for 4478223 soon. More testing on Linux and Win32 will be done to see if there're any other problem with the fix for 4478223 and the fix described above in place. ###@###.### 2001-09-26 Summary on the status on this bug: This bug involves fixes in 4 different areas and two of them are described above (one in ObjectMonitor::raw_enter and the other is in hprof). The other two areas being fixed by Dave Cox in C1 are: 4478223 2/3 "oopmap not found" assertion failure in jvmpi heap dump collection Compiler1 now uses a frameless stub to call SharedRuntime::jvmpi_method_exit, which may block, so that the compiled call site doesn't need an oop map and cannot be deoptimized. If the caller is returning an oop or rethrowing and exception, that oop is preserved in its thread's vm_result field. 4505853 3/3 JVMPI method enter event not always posted by compiled method If a C1-compiled synchronized method was deoptimized after calling monitorenter upon entry, jvmpi_method_entry was not called. This is fixed by using a special monitorenter stub for the SynchronizationEntryBCI when JVMPI method entry events are enabled and performing the JVMPI notification from the stub if its caller has been deoptimized. These fixes will be available in build 81. Here are the testing results on various platforms with my workspace containing all these fixes testing the following hprof options: -Xrunhprof:cpu=times -Xrunhprof:cpu=samples -Xrunhprof:heap=dump -Xrunhprof:heap=all 1) Solaris Sparc I ran the above hprof option on Jmol, TeaTimeJ, mailpuccino, and SwingSet2. All except -Xrunhprof:heap=[dump,all] work fine. For -Xrunhprof:heap=[dump,all], we are getting intermittent errors and empty heap dump when running hprof on SwingSet2. One error is OutOfMemoryError exception during shutdown and thus no heap dump. The other error is HPROF ERROR: heap dump size < 0. A new bug 4507533 is filed for this heap dump problem. 2) WinNT Same result as Solaris Sparc. 3) Linux It works fine with Jmol, TeaTimeJ and mailpuccino benchmark but I also got the heap dump size < 0 error. When running -Xrunhprof:cpu=[times,samples] on SwingSet2, the VM hangs or crashes. A new bug 4507553 is filed for the linux hang problem.
11-06-2004
PUBLIC COMMENTS .
10-06-2004
SUGGESTED FIX ###@###.### 2001-09-24 See attached webrev.tar.Z
24-09-2001