It seems like when Java heap is exhausted, G1 does not let native code owning a critical region go before throwing OOME from Java code. This is a regression from JDK6.
Please see comments for more details.
OOM can also cause crash with following message:
# Out of Memory Error (allocation.cpp:211), pid=17209, tid=2364230560
# JRE version: 7.0-b126
# Java VM: Java HotSpot(TM) Server VM (20.0-b06 compiled mode linux-x86 )
--------------- T H R E A D ---------------
Current thread (0x08492400): JavaThread "Thread-0" [_thread_in_vm, id=17233, stack(0x8ce64000,0x8ceb5000)]
Stack: [0x8ce64000,0x8ceb5000], sp=0x8ceb3a20, free space=318k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x686e1f]# Host info: Linux vmsqe-p4-14 18.104.22.168-0.42.10-smp #1 SMP Tue Apr 27 05:11:27 UTC 2010 i686 i686 i386 GNU/Linux