JDK-4386959 : Exceptions in JVM/Native code or OutOFMemoryError using 1.3.0_01/1.3.0
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 1.3.0
  • Priority: P2
  • Status: Closed
  • Resolution: Cannot Reproduce
  • OS: windows_nt
  • CPU: x86
  • Submitted: 2000-11-08
  • Updated: 2001-09-14
  • Resolved: 2001-09-14
Related Reports
Relates :  
JDK-4473322 - (cs) Intermittent failures in CharsetDecoder
Description
Customer Problem description:
-----------------------------

After running through many iterations of our routines, something bad eventually happens:
1. out of memory error, java.exe exits
2. java.exe crashes, in native code
3. jvm exception, java.exe exits

We're running the hotspot client vm:
java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Client VM (build 1.3.0-C, mixed mode)



1. out of memory:

E:\test\java>java -classpath dialogserver.jar;parser.jar;servlet.jar;jaxp.jar;.
-Detc=. -Xms160M -Xmx160M EngineLoadTest2 files\demos\mutuallife\mlife.htm -a -t
8 -i-1
Input:          files\demos\mutuallife\mlife.htm
Output:         none
Caching:        true
Browser:        IE5
Iterations:     -1
Threads:        8
Compression:    false
Inline JS:      true
TerseOutput:    false
File Size:      263,447 bytes

Computed Adler = 6e87d7f3

Press Enter to Stop

          Pages        Seconds      Pages/Sec
         29,793            928             32java.lang.OutOfMemoryError
        <<no stack trace available>>

E:\test\java>

2. java.exe crash
[GC 28602K->27230K(32576K), 0.0192270 secs]
[GC 29328K->27573K(32576K), 0.0113367 secs]
[GC 29685K->28110K(32576K), 0.0157567 secs]
[GC 30217K->28658K(32576K), 0.0155967 secs]
[Full GC 30768K->12316K(32576K), 0.5985409 secs]
# 
# An EXCEPTION_ACCESS_VIOLATION exception has been detected in native code 
outside the VM.
# Program counter=0x103
#
        965,068         19,365             55
        

3. jvm exception

C:\test\java>java -classpath dialogserver.jar;parser.jar;servlet.jar;. -Detc=c:test\java -Xms128M -Xmx128M EngineLoadTest2 dialogserver/demos/mutuallife/mlife.
htm -i-1 -t3 -a
Input:          dialogserver/demos/mutuallife/mlife.htm
Output:         none
Caching:        true
Browser:        IE5
Iterations:     -1
Threads:        3
Compression:    false
Inline JS:      false
TerseOutput:    true
File Size:      111,368 bytes

Computed Adler = a3932c78

Press Enter to Stop

          Pages        Seconds      Pages/Sec
        125,587          7,307             12#
# HotSpot Virtual Machine Error, EXCEPTION_ACCESS_VIOLATION
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Error ID: 4F533F57494E13120E43505002D4
#

abnormal program termination

C:\test\java>


Similar behaviour is seen with 1.3.0_01 as well:

C:\TEMP>runtest2

C:\TEMP>jrcs -d -t8 -i-1

C:\TEMP>call jr dialogserver\samples\colorshape\colorshape.htm -d -t8 -i-1


C:\TEMP>java -showversion -classpath dialogserver.jar;parser.jar;servlet.jar;. -
Detc=. -Xms32M -Xmx32M EngineLoadTest2 dialogserver\samples\colorshape\colorshap
e.htm -d -t8 -i-1
java version "1.3.0_01-beta"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0_01-beta)
Java HotSpot(TM) Client VM (build 1.3.0-C, mixed mode)

Input:          dialogserver\samples\colorshape\colorshape.htm
Output:         none
Caching:        false
System.gc():    0
Browser:        IE5
Iterations:     -1
Threads:        8
Compression:    true
Inline JS:      false
TerseOutput:    true
File Size:      2,481 bytes

Press Enter to Stop

          Pages        Seconds      Pages/Sec
            629             27             25#
# HotSpot Virtual Machine Error, EXCEPTION_ACCESS_VIOLATION
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Error ID: 4F533F57494E13120E43505002D4
#

abnormal program termination


Unfortunately, we can't 
duplicate the problem with 100% consistency.  However, we are often able 
to get some type of problem to occur within a few hours of run time. Also, 
I'm fairly certain that increasing the heap size will just delay the 
problem  We first detected the problem with a heap size of -Xms256M 
-Xmx256M.  It was a slightly different test, but it ran for 1-2 days and 
processed between 4 and 16 million pages before crashing the jvm.

Attached is a self-extracting zip that should include all the files you 
need.  Extract it to an empty directory, and run the runtest.bat file you 
find there.  It should crash or run out of memory after running a while.

Info on the two machines we've seen the problems on are also attached. 
They're reports from the Windows NT Diagnostics utility. 

Name: yyT116575			Date: 01/09/2001


C:\>java -version
java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Client VM (build 1.3.0-C, mixed mode)


Test Platform: Dell Inspiron 7000 PentiumII 400, 256MB ram,
running JVM 1.3 with vmargs: -mx8m  (purposely starting small)

Using OptimizeIt 4.01 Pro (build 413) Intuit Systems www.optimizeit.com
to track down memory leak.  Running in conjunction with JSW 2.0.
Selected "reference graph" option.  After lengthy (2 minute pause) with CPU at
100% usage, following error produced:

This can be repeated upon demand.

javawebserver: abnormal program termination
javawebserver: OptimizeIt generic Audit System. [Hotspot runtime detected]
javawebserver: #
javawebserver: # HotSpot Virtual Machine Error, EXCEPTION_ACCESS_VIOLATION
javawebserver: # Please report this error at
javawebserver: # http://java.sun.com/cgi-bin/bugreport.cgi
javawebserver: #
javawebserver: # Error ID: 4F533F57494E13120E43505002D4
javawebserver: #
javawebserver: High-resolution counter available
javawebserver: C:\JavaWebServer2.0\bin\..\bin\jserv.exe
javawebserver: C:\JAVAWE~1.0\bin\..\bin\jserv.exe
javawebserver: Program C:\JDK1.3\bin\java.exe exited with code 3
Java Web Server WARNING: Server javawebserver unexpectedly exited with code 3
(Review ID: 114819)
======================================================================

Name: yyT116575			Date: 01/17/2001


java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Client VM (build 1.3.0-C, mixed mode)


HotSpot gives me random memory weirdness -- first, using OptimizeIt 4.0 and comparing it with the Windows NT task manager, I can see that, for example, the VM has allocated 200+ megs of RAM. However, OptimizeIt reports that the java heap is roughly 20-40 megabytes.

Sometimes, HotSpot crashes with an error such as the following (which is an actual error message that I have just received):

#
# HotSpot Virtual Machine Error, EXCEPTION_ACCESS_VIOLATION
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Error ID: 4F533F57494E13120E43505002D4
#

abnormal program termination

This happens on Windows NT 4.0, service pack 6a, although we are getting the same type of weirdness on other types of Windows systems.
(Review ID: 115271)
======================================================================
Comments
EVALUATION Currently attempting to reproduce the failure in-house with a debug version of the 1.3.0 HotSpot Client VM. On a dual-CPU NT 4.0 machine, has so far run to over 750,000 pages with -verbose:gc turned on. In addition to trying to reproduce the failure with 1.3.0, we would like to ship the customer an early access version of 1.3.1, in which some bugs were fixed in both the run-time system as well as the client compiler. Have left messages with the Sun contact for this bug; if customer would leave an email address somewhere in the bug comments could proceed with this. kenneth.russell@eng 2000-12-08 Have reproduced the failure with the debug version of the 1.3.0 HotSpot Client VM after generation of 1.6 million pages over a few days. Assertion failure is in handles.cpp, line 24; apparently trying to make a handle for a bad object. A couple of places where this could have happened have been fixed in 1.3.1; will re-run with the debug version of the 1.3.1 VM. kenneth.russell@eng 2000-12-11 After sending the customer the 1.3.1 VM, received an OutOfMemoryError stack trace from the customer. Testing internally indicates strongly that this problem is due to the customer's relying upon finalization to close GZIPOutputStreams instead of closing them explicitly. Have suggested to the customer the necessary code modifications. Also re-running above test with 1.3.1 VM to see whether crash seen earlier has been fixed. kenneth.russell@eng 2000-12-12 Ran customer's test against a C1 jvm_g.dll in the 1.3.1 libraries, running java_g within the Visual Studio IDE to be able to stop upon access violations. After 1.5 days and probably between 1.3 and 2.0 million web pages generated, the VM crashed with hundreds of the following messages in the IDE's status window: "First-chance exception in java_g.exe: 0xC000001D: Illegal Instruction." Unfortunately the IDE did not keep the process around and we were therefore not able to obtain a stack trace. Need to think some more about how to get the VM to suspend itself upon this kind of failure. Putting in an infinite loop in os_win32.cpp, os::message_box, may be the best solution. kenneth.russell@eng 2000-12-15 Received the following email from the customer: > I did this, and it didn't help. In fact, I was already calling the > GZIPOutputStream's finish() method, which is essentially what close() > does, but without closing the contained output stream. (I'm getting this > info by looking at the source for GZIPOutputStream that comes with the 1.3 > SDK.) > > Neither of these methods causes the native C-heap memory to be freed, > because they don't call def.end(), where def is the contained Deflater > object that controls the C-heap memory. > > I think the solution might be to derive a class from GZIPOutputStream that > adds a method to call def.end(), e.g. > > public class MyGZIPOutputStream extends GZIPOutputStream { > // constructors... > > public void killDeflater() > { > def.end(); > } > } > > Does this seem like it would work? Do you see any problems with a > solution like this? Is there a simpler solution? I believe this is the correct workaround for the OutOfMemoryError. Awaiting customer input to learn which is the showstopper problem for the customer, the OutOfMemoryError or the VM crash. It looks like these are two independent problems and I have recommended opening a new bug against the libraries for the GZIPOutputStream problem with the customer's above suggested workaround contained within. Again, so far we have not been able to get a stack trace for the VM crash. kenneth.russell@eng 2001-01-03 A follow-up email from the customer (not in response to the above question): > I gave this a try, and it doesn't seem to fix the problem - I still get > java.lang.OutOfMemory errors. However, it does seem to have some effect - > if I look at the memory usage in the task manager, the version that calls > def.end() has a basically flat memory usage, whereas the version without > it has a sawtooth look, probably caused by the deflater native memory > being used, and eventually garbage collected. > > I hope this information helps. Do you have any other ideas for things I > can try to fix this problem? The OutOfMemoryErrors are almost certainly being caused by running out of file descriptors. Calling the end() method of the deflater should not change the program behavior significantly. Please ask the customer to verify their application calls the close() method of all OutputStreams before releasing their last references. kenneth.russell@eng 2001-01-03 The recent update to the bug description did not follow the instructions from my earlier email: 1. Please run the test with java -classic 2. If that fails with an OutOfMemoryError, run OptimizeIt against either a JDK 1.3 "java -classic" VM or against the 1.2.2 VM (which did not have HotSpot) JVMPI was not stable in HotSpot in 1.3.0 but will be better in 1.3.1. kenneth.russell@eng 2001-01-09 The version of the customer's application with the calls to Deflater.end() runs fine on JDK 1.3.1 on a dual-CPU 667 MHz PIII running NT 4.0 with -Xms160M -Xmx160M; increased the script to generate 40,000 pages with no problems. It also works if the CPU affinity for java.exe is set to CPU 0, which should emulate a single-CPU machine. Ran the application under OptimizeIt 4.0; Strings and char arrays consume the majority of the heap. However, a substantial number of finalizable objects are being allocated in MergeThread.run(). This is likely the cause of the OutOfMemoryError. Finalizers are never necessary unless native code is involved. They are especially problematic because they require two full GCs before they can be reclaimed. It is not clear without the source code what those finalizable objects are, but eliminating their allocation over time should solve the problem. Please read the following paper for more information: Automatic Memory Management for the JavaTM Programming Language http://jsp.java.sun.com/javaone/javaone2000/pdfs/TS-1172.pdf kenneth.russell@eng 2001-02-09 (What I meant by "eliminate their allocation over time" above was to re-use some of these finalizable objects, if possible, instead of discarding them and creating new ones.) Have run the repeated 10,000-page-generation test (which, note, creates a new VM after each run of the test) with the version of the customer's jar which closes all GZIPOutputStreams (avoiding exhausting physical memory) on a 192 MB dual-CPU PIII 667 MHz, with one CPU disabled at the BIOS level, on a recent build of JDK 1.3.1, with -Xms160m -Xmx160m, for 24 hours with no OutOfMemoryErrors generated. At this point I do not think the VM's finalization mechanism is the cause of the problem, although I could be wrong (perhaps the multiprocessor kernel still has different scheduling behavior than the single-processor kernel even if one CPU is disabled). Instead it looks to me like the OutOfMemoryErrors reported by the customer are caused by the machine's running out of swap space. Using the 1.3.1 VM, have not seen an occurrence of the reported crash in quite a while (as confirmed by the customer.) kenneth.russell@eng 2001-02-21 I just tried ten runs each of 5000 page tests on win32_i486, solsparc, and linux_i386 with JDK-1.4.0-beta_refresh-b68, and they all passed. peter.kessler@Eng 2001-06-22
21-02-2001