JDK-4855795 : Spurious OutOfMemoryError exceptions
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 1.4.2
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • OS: windows_2000
  • CPU: x86
  • Submitted: 2003-04-29
  • Updated: 2008-11-05
  • Resolved: 2006-01-31
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other JDK 6
1.4.2_13Fixed 6 b70Fixed
Related Reports
Relates :  
Relates :  
Name: rmT116609			Date: 04/29/2003

java version "1.4.1_01"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_01-b01)
Java HotSpot(TM) Client VM (build 1.4.1_01-b01, mixed mode)

java version "1.4.2-beta"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2-beta-b19)
Java HotSpot(TM) Client VM (build 1.4.2-beta-b19, mixed mode)

Microsoft Windows 2000 [Version 5.00.2195]

The HotSpot VM sometimes throws an OutOfMemoryError even
though there is plenty of reclaimable garbage present. It
appears that an allocation failure on an application
request does trigger a full garbage collection, but an
OutOfMemoryError may be thrown regardless of the results of
that GC.

In the process of debugging an out-of-memory exit in a
complex application, I wrote a simple test program that
allocates a series of 4KB blocks (byte arrays). When the
amount of free space on the heap gets small, say <1MB free,
the program begins dropping its references to some of the
blocks & continues to allocate new blocks. Depending on
various heap option settings, an OutOfMemory exception may
be thrown. An example of the output in one of these cases

C:\heaptest>java -server -verbose:gc -XX:NewRatio=4 -x100m -
Xms100m HeapTest

requested allocation size = 4096  actual size = 4112 bytes
[Full GC 446K->330K(100352K), 0.0104416 secs]
--- starting allocations -------
[GC 16714K->16714K(100352K), 0.0470250 secs]
[GC 33097K->33097K(100352K), 0.0497818 secs]
[GC 49481K->49481K(100352K), 0.0480004 secs]
[GC 65865K->65865K(100352K), 0.0489293 secs]
[Full GC 82249K->82249K(100352K), 0.0753784 secs]
24908: may throw OutOfMemory. free heap is 4632
[Full GC 100348K->98307K(100352K), 0.0306476 secs]
[Full GC 98307K->98307K(100352K), 0.1124120 secs]

24908: *** Out of Memory Exception caught *** (expected
2032K garbage)
 free heap is  2092448 bytes
 total blocks allocated = 24399

The line beginning 24908: is logged indicating that the
free heap space (as returned by Runtime.freeMemory()) is
low enough that the next 4KB allocation attempt may fail.
However, at this point there are roughly 2MB of previously
allocated but now unreferenced blocks in the heap. The next
line shows a full GC, presumably caused by the allocation
  [Full GC 100348K->98307K(100352K), 0.0306476 secs]
Note that this full GC reclaimed around 2MB, yet the
OutOfMemoryError exception is still thrown. Furthermore,
when the exception is caught, freeMemory() shows that 2MB
are indeed available. Also, immediately retrying the same
allocation now succeeds!

This behavior is so unexpected that I do not see how it
could be considered anything but a bug. I realize that the
VM spec is vague on this subject, but the relevant sentence

"If a computation requires more heap than can be made
available by the automatic storage management system, the
Java virtual machine throws an OutOfMemoryError."

Clearly in the example above, the system COULD have made
sufficient heap available had it tried to do so

Lest you think this is a contrived example, the original
problem causing our application to crash exhibited similar
symptoms. We were able to prevent most of the application
errors by periodically forcing a full GC via System.gc(),
however this is not recommended & should not be necessary.

1. compile the attached HeapTest.java class
2.java -server -verbose:gc -XX:NewRatio=3 -Xmx100m -Xms100m HeapTest
(different options may or may not reproduce the problem)

This bug can be reproduced always.

---------- BEGIN SOURCE ----------
class HeapTest
    static final private int KB = 1024;
    static final private int MB = 1024*1024;

    public static void main( String[] args )
        Runtime rt = Runtime.getRuntime();

        java.util.Vector v = new java.util.Vector(50000);

        int MAXITER = 2000000;
        int BLOCKSIZE = 4*KB;
        int actBlockSize = BLOCKSIZE;

        // determine actual space allocated per block including overhead
        long preFree = rt.freeMemory();
        byte[] ba = new byte[BLOCKSIZE];
        long curFree = rt.freeMemory();
        ba = null;
        actBlockSize = (int)(preFree-curFree);
        System.out.println( "requested allocation size = " + BLOCKSIZE
                                + "  actual size = " + actBlockSize + "
bytes" );

        System.out.println( "--- starting allocations -------" );

        int  gCount = 0;

        for( int i=1; i < MAXITER; i++ ) {
            preFree = rt.freeMemory();

            if( preFree < 2*BLOCKSIZE ) {
                System.out.println( i + ": may cause allocation failure. free
heap is " + preFree );
             //   System.out.println( i + ": forcing GC" );
             //   System.gc();

            try {
                ba = new byte[BLOCKSIZE];
                v.add( ba );
                ba = null;
            catch( java.lang.OutOfMemoryError x ) {
                System.out.println( "" );
                System.out.println( i + ": *** Out of Memory Exception caught
*** (expected " + (gCount*BLOCKSIZE/KB) + "K garbage)" );
                System.out.println( " free heap is now " + rt.freeMemory() + "
bytes" );
                System.out.println( " total blocks allocated = " + v.size() );
                System.out.println( "" );

            curFree = rt.freeMemory();

            long freed = (curFree - preFree) + actBlockSize;
            if( freed > 0 ) {
                System.out.println( i + ": GC freed " + freed/KB + "K"
                    + " (" + (gCount*actBlockSize/KB) + "K expected)" );
                gCount = 0;

            // drop references to the most recently allocated 2 blocks if free
memory is tight
            if( rt.freeMemory() < 1*MB && v.size() > 2 ) {
                v.removeElementAt( v.size()-1 );
                v.removeElementAt( v.size()-1 );
                gCount += 2;

---------- END SOURCE ----------
(Review ID: 181286) 

SUGGESTED FIX Event: putback-to Parent workspace: /net/jano.sfbay/export/disk05/hotspot/ws/main/gc_baseline (jano.sfbay:/export/disk05/hotspot/ws/main/gc_baseline) Child workspace: /net/prt-web.sfbay/prt-workspaces/20060126152955.ysr.policy/workspace (prt-web:/net/prt-web.sfbay/prt-workspaces/20060126152955.ysr.policy/workspace) User: ysr Comment: --------------------------------------------------------- Job ID: 20060126152955.ysr.policy Original workspace: neeraja:/net/spot/scratch/ysr/policy Submitter: ysr Archived data: /net/prt-archiver.sfbay/data/archived_workspaces/main/gc_baseline/2006/20060126152955.ysr.policy/ Webrev: http://analemma.sfbay.sun.com/net/prt-archiver.sfbay/data/archived_workspaces/main/gc_baseline/2006/20060126152955.ysr.policy/workspace/webrevs/webrev-2006.01.26/index.html Fixed 6369448: Unnecessary FullGC seems to run in 6.0b65 Fixed 4855795: Spurious OutOfMemoryError exceptions Webrev: http://analemma.sfbay/net/spot/scratch/ysr/policy/webrev The problem in 6369448 was that the policy for deciding whether it was possible to allocate an object in the young gen was based on the maximum potential size of young gen (not even Eden) while there was nothing in the resizing policy that was affected by the failure to accomodate such requests. The correct thing, under the circumstances, is to recognize that Eden cannot allocate objects larger than its _current_ size, and allow an older generation to satisfy such an allocation request if possible. This is a day-one bug in the policy object. The problem in 4855795 was identified by Peter, as described in the bug report. We now allow the VM thread to allocate out of from space when space is tight. This prevents the "unexpected" out of memory error described. Piggyback: recent testing with -XX:+CMSMarkStackOverflowALot revealed some obsolete non-product code had been left around. The lock in question had been removed sometime ago, but not all of its uses had been removed. Some (related) documentation comments were also fixed. Reviewed by: Jon Masamitsu, John Coomes Approved (low risk) by: Dave Cox Fix Verified: yes Verification Testing: 6369448 test in bug report 4855795 test in bug report Other testing: specjvm, PRT refworkload (performance) with CMS, Serial. Files: update: src/share/vm/memory/collectorPolicy.cpp update: src/share/vm/memory/concurrentMarkSweepGeneration.cpp update: src/share/vm/memory/defNewGeneration.cpp Examined files: 3768 Contents Summary: 3 update 3765 no action (unchanged)

SUGGESTED FIX ------- defNewGeneration.cpp ------- 418c418,420 < if (Heap_lock->owned_by_self()) { --- > if (Heap_lock->owned_by_self() || > (SafepointSynchronize::is_at_safepoint() && > Thread::current()->is_VM_thread())) {

EVALUATION This looks like it might be a corner case in our handling of the nearly out of memory case. Usually, TwoGenerationCollectorPolicy::mem_allocate_work will make two attempts to allocate in the young generation: one without the heap lock, which can't allocate in from-space, and one with the heap lock, which then can allocate in from-space. In contrast, I think GenCollectedHeap::do_collection just calls gen->allocate, but the requesting thread owns the heap lock, not the VM thread, so we don't attempt to allocate in the from-space. That said, users really shouldn't be depending on Runtime.freeMemory() to know when memory is free. For instance, because of fragmentation, just because Runtime.freeMemory() says there are 4KB free doesn't mean that the user can get all 4KB for one object. That and the fact that Runtime.freeMemory() is notoriously broken, inaccurate, and untimely. ###@###.### 2003-05-01 ----------------------------------------------------------------

SUGGESTED FIX It might be enough for DefNewGeneration::allocate_from_space to allow allocations from from-space if we are the VM thread. The problem will be convincing ourselves that that doesn't cause a race in which two threads think they can allocate from from-space: one because they hold the heap lock, and one because they are the VM thread. That would hand out the same address as two different objects, and be hard to diagnose. ###@###.### 2003-05-01 ----------------------------------------------------------------