The out of swap space hs_err output looks like:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Out of swap space to map in thread stack.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (/tmp/jprt/P1/B/215741.dcubed/source/src/os_cpu/solaris_sparc/vm/os_solaris_sparc.cpp:592), pid=9203, tid=2
#
# JRE version: 7.0-b125
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.0-b07-internal-201101192157.dcubed.7012505_for_hsx_20-fastdebug mixed mode solaris-sparc )

---------------  T H R E A D  ---------------

Current thread (0x0000000100130000):  JavaThread "main" [_thread_in_vm, id=2, stack(0xffffffff7b000000,0xffffffff7b100000)]

Stack: [0xffffffff7b000000,0xffffffff7b100000],  sp=0xffffffff7b0fe030,  free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x126c7f0]
[error occurred during error reporting (printing native stack), id 0xe0000000]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  java.lang.Thread.start0()V
J  java.lang.Thread.start()V
J  nsk.jvmti.ResourceExhausted.resexhausted001.run([Ljava/lang/String;Ljava/io/PrintStream;)I
j  nsk.jvmti.ResourceExhausted.resexhausted001.main([Ljava/lang/String;)V+9
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x000000011574b800 JavaThread "fleece" [_thread_in_native, id=35416, stack(0xffffffee0e600000,0xffffffee0e700000)]

<snip the huge number of "fleece" threads>
The malloc failure hs_err output looks like:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32764 bytes for ChunkPool::allocate
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (/tmp/jprt/P1/B/215741.dcubed/source/src/share/vm/memory/allocation.cpp:211), pid=12781, tid=7
#
# JRE version: 7.0-b125
# Java VM: Java HotSpot(TM) Server VM (20.0-b07-internal-201101192157.dcubed.7012505_for_hsx_20-fastdebug compiled mode solaris-sparc )

---------------  T H R E A D  ---------------

Current thread (0x00183400):  VMThread [stack: 0xd6700000,0xd6780000] [id=7]

Stack: [0xd6700000,0xd6780000],  sp=0xd677f480,  free space=509k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x11589bc]# Host info: SunOS vm-v440-01 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V440
This bug is for the following VM/NSK test:

    nsk/jvmti/ResourceExhausted/resexhausted001

The following note documents issues with resexhausted001 that
were found in 2011.01.19 RT_Baseline nightly testing:

6 resexhausted001 failed in
   linux-amd64_vm_server_mixed_nsk.jvmti.testlist
       - timeout on machine vm-v20z-2
   windows-amd64_vm_server_comp_nsk.jvmti.testlist
       - timeout on machine jck-amd2
   solaris-sparc_vm_server_comp_nsk.jvmti.testlist
       - malloc failed on machine vm-v440-01
   linux-i586_vm_client_mixed_nsk.jvmti.testlist
       - exit code 139 on machine vm-v20z-34
   windows-amd64_vm_server_mixed_nsk.jvmti.testlist
       - timeout on machine jck-amd2
   solaris-sparcv9_vm_server_mixed_nsk.jvmti.testlist
       - out of swap space on machine soda

   resexhausted001 was previous covered by the following bug:
       6302804 3/1 Hotspot VM dies ungraceful death when C heap is
                   exhausted in various places.

   Coleen used 6302804 to improve the way these types of resource
   exhaustion failures are reported. The two Solaris failures
   above are good examples of the new diagnostic output.

   Coleen, Nicolay and I are discussing the resexhausted00[1-4]
   tests and are working out the type of bug to file to cover
   the failure modes that we are observing. 

Here are the URLs for the failing configs:

timeout on machine vm-v20z-2:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/linux-amd64/server/mixed/linux-amd64_vm_server_mixed_nsk.jvmti.testlist/analysis.html

timeout on machine jck-amd2:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/windows-amd64/server/comp/windows-amd64_vm_server_comp_nsk.jvmti.testlist/analysis.html

malloc failed on machine vm-v440-01:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/solaris-sparc/server/comp/solaris-sparc_vm_server_comp_nsk.jvmti.testlist/analysis.html

exit code 139 on machine vm-v20z-34:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/linux-i586/client/mixed/linux-i586_vm_client_mixed_nsk.jvmti.testlist/analysis.html

timeout on machine jck-amd2

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/windows-amd64/server/mixed/windows-amd64_vm_server_mixed_nsk.jvmti.testlist/analysis.html

out of swap space on machine soda:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/solaris-sparcv9/server/mixed/solaris-sparcv9_vm_server_mixed_nsk.jvmti.testlist/analysis.html
The following VM/NSK test also failed in the 2011.01.19 RT_Baseline
nightly with a malloc failure:

    nsk/jvmti/ResourceExhausted/resexhausted004

This particular failure didn't stand out in my initial analysis
because it is on the known fail_list for a different bug (6606767).

Here is the URL for the failing config:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/solaris-sparc/server/comp/solaris-sparc_vm_server_comp_nsk.jvmti.testlist/analysis.html

And here is the failing hs_err output:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32764 bytes for ChunkPool::allocate
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (/tmp/jprt/P1/B/215741.dcubed/source/src/share/vm/memory/allocation.cpp:211), pid=13230, tid=7
#
# JRE version: 7.0-b125
# Java VM: Java HotSpot(TM) Server VM (20.0-b07-internal-201101192157.dcubed.7012505_for_hsx_20-fastdebug compiled mode solaris-sparc )

---------------  T H R E A D  ---------------

Current thread (0x00183400):  VMThread [stack: 0xf8900000,0xf8980000] [id=7]

Stack: [0xf8900000,0xf8980000],  sp=0xf897ee80,  free space=507k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x11589bc]# Host info: SunOS vm-v440-01 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V440
This bug is primarily used to track malloc failures due to excessive thread creation not general malloc failures.  The new output tells you not to file a bug for those because it is generally a system problem that is the cause.  The case with excessive thread creation should be handled so it doesn't fail with malloc failures if possible.

Timeouts may not be able to be handled, only error return from thread create.  On some systems, the os tries to wait for memory to be available or tries to swap out other processes if there's sufficient swap space (ie I can't get this to fail on some systems).