We are running the tests with a JDK patch for issue 6826736. 

# /space/system-test/tools/jdk1.6.0_14-64/bin/java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
OpenJDK 64-Bit Server VM (build 16.0-b05-2009-07-22-225507.kvn.6826736, mixed mode)

These are the JDK options used

        <jvm-options>-XX:MaxTenuringThreshold=0</jvm-options>
        <jvm-options>-XX:SurvivorRatio=128</jvm-options>
        <jvm-options>-XX:+DisableExplicitGC</jvm-options> 
        <jvm-options>-XX:PermSize=64m</jvm-options>
        <jvm-options>-XX:MaxPermSize=128m</jvm-options>
        <jvm-options>-XX:+UseParNewGC</jvm-options>
        <jvm-options>-XX:ParallelGCThreads=7</jvm-options>
        <jvm-options>-XX:+UseConcMarkSweepGC</jvm-options>
        <jvm-options>-XX:CMSInitiatingOccupancyFraction=1</jvm-options>
        <jvm-options>-XX:LargePageSizeInBytes=2m</jvm-options>
        <jvm-options>-XX:+PrintGCDetails</jvm-options>
        <jvm-options>-XX:+PrintGCDateStamps</jvm-options>
        <jvm-options>-XX:+PrintClassHistogram</jvm-options>
        <jvm-options>-XX:+HeapDumpOnOutOfMemoryError</jvm-options>
        <jvm-options>-Xloggc:${com.sun.aas.instanceRoot}/logs/gc.log</jvm-options>  
        <jvm-options>-Dorg.jvnet.glassfish.comms.sip.timer.queues=10</jvm-options>
        <jvm-options>-XX:ErrorFile=st_hs_err_pid%p.log</jvm-options>
        <jvm-options>-server</jvm-options> 
        <jvm-options>-Xss128k</jvm-options>
        <jvm-options>-Xmn128m</jvm-options>
        <jvm-options>-Xms6000m</jvm-options> 
        <jvm-options>-Xmx6000m</jvm-options>
        <jvm-options>-XX:+UseCompressedOops</jvm-options>

About 15 - 20 minutes in the test an instance crashes. After this another 4 instance crashed one after the other. The logs are uploaded at this location.

http://agni-1.sfbay.sun.com/net/asqe-logs/export1/SailFin/Results/ST/2.0/b27/vmcrash/