FULL PRODUCT VERSION :
1.6.0_07 64-bit

FULL OS VERSION :
Linux Red Hat RHEL 4

A DESCRIPTION OF THE PROBLEM :
When running the attached sample code the JVM crashes with this error message:

java.lang.OutOfMemoryError: requested -1610612736 bytes for GrET in /BUILD_AREA/jdk6_07/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Compile the attached code then run it as

java -Xmx15g -Xms15g -verbose:gc Mark 125000000 10000000

EXPECTED VERSUS ACTUAL BEHAVIOR :
The program should run to completion but instead it produces this output:

[Full GC#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested -1610612736 bytes for GrET in /BUILD_AREA/jdk6_07/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?
#
#  Internal Error (allocation.inline.hpp:42), pid=11989, tid=1081280864
#  Error: GrET in /BUILD_AREA/jdk6_07/hotspot/src/share/vm/utilities/growableArray.cpp
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode linux-amd64)
# An error report file with more information is saved as:
# /home/murrap/hs_err_pid11989.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#
Aborted

ERROR MESSAGES/STACK TRACES THAT OCCUR :
[Full GC#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested -1610612736 bytes for GrET in /BUILD_AREA/jdk6_07/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?
#
#  Internal Error (allocation.inline.hpp:42), pid=11989, tid=1081280864
#  Error: GrET in /BUILD_AREA/jdk6_07/hotspot/src/share/vm/utilities/growableArray.cpp
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode linux-amd64)
# An error report file with more information is saved as:
# /home/murrap/hs_err_pid11989.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#
Aborted


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.*;

class Foo implements Serializable {
        static int counter=0;
        private int hash;

        static synchronized int nextCount() {
                return counter++;
        }

        public Foo() {
                this.hash = nextCount();
        }

        public int hashCode() {
                return hash;
        }

        public boolean equals(Object o) {
                if (o == this)
                        return true;
                return false;
        }
}

public class Mark extends Thread {
        static int arrSz;
        static int strArrSz;

        public void run() {
                int total = 0;
                Foo[] arr = new Foo[arrSz];
                String[] arr2 = new String[strArrSz];
                System.out.println("1");
                for (int i=0; i<arrSz; i++) {
                        arr[i] = new Foo();
                        // arr[i].hashCode();
                        System.identityHashCode(arr[i]);
                }
                System.out.println("2");

                for (int i=0; i<arrSz; i++) {
                        String foo = new String(""+i);
                        arr2[i%strArrSz] = foo;
                        if (arr[i].equals(arr2[i%strArrSz])) {
                                System.out.println("Foo!");
                        }
                }
                System.out.println("3");
        }

        public static void main(String[] args) throws Exception {
                arrSz = Integer.parseInt(args[0]);
                strArrSz = Integer.parseInt(args[1]);
                new Mark().start();
                new Mark().run();
        }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
No workaround except to fix signed integer overflow problem in mark stack code.

The error was triggered by serialization of a large in-memory cache. The cache is persisted to disk but the objects continue to be live for the remainder of the uptime of the application.

The fundamental problem is that serialization calls System.identityHashCode() on every object to be serialized as part of the creating the Handle hashmap used to check for re-serialization of previously serialized objects.

As a consequence, once the cache is serialized we get stability problems. Even when (for slightly smaller collections) it doesn't cause a crash we see huge increases in Full GC time and transient (native) memory utilization due to the overhead of storing off and restoring all the identity hashes from the mark words.