This assertion failure was observed during some ad-hoc testing of G1. The test directory can be found at
 http://sqeweb.sfbay.sun.com/nfs/tools/gtee/results/JDK6/ADHOC/VM/2010-04-13_07/vm/solaris-i586/server/mixed/solaris-i586_server_mixed_vm.gc.testlist/ResultDir/JumbleGC002/

The machine on which the test ran is vm-b1600x-2, a single cpu intel x86 machine:

vm-b1600x-2{jc234399}:263> uname -a
SunOS vm-b1600x-2 5.10 Generic_139556-08 i86pc i386 i86pc
vm-b1600x-2{jc234399}:264> psrinfo -v
Status of virtual processor 0 as of: 04/21/2010 11:43:59
  on-line since 11/15/2009 14:52:04.
  The i386 processor operates at 1533 MHz,
        and has an i387 compatible floating point processor.

The stack trace (from the hs_err file) is:

Stack: 
[error occurred during error reporting (printing stack bounds), id 0xe0000000]

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x176faaa];;  __1cHVMErrorGreport6MpnMoutputStream__v_+0x5be
V  [libjvm.so+0x1770c32];;  __1cHVMErrorOreport_and_die6M_v_+0x586
V  [libjvm.so+0x7f24b9];;  __1cYreport_assertion_failure6Fpkci1_v_+0x61
V  [libjvm.so+0x1770586];;  __1cHVMErrorGreport6MpnMoutputStream__v_+0x109a
V  [libjvm.so+0x1770c32];;  __1cHVMErrorOreport_and_die6M_v_+0x586
V  [libjvm.so+0x7f24b9];;  __1cYreport_assertion_failure6Fpkci1_v_+0x61
V  [libjvm.so+0x8cb9b0];;  __1cFframebDinterpreter_frame_monitor_end6kM_pnPBasicObjectLock__+0x70
V  [libjvm.so+0x8c5a32];;  __1cFframeToops_interpreted_do6MpnKOopClosure_pknLRegisterMap_b_v_+0x85e
V  [libjvm.so+0x8c92d0];;  __1cFframeQoops_do_internal6MpnKOopClosure_pnPCodeBlobClosure_pnLRegisterMap_b_v_+0xb8
V  [libjvm.so+0x1683244];;  __1cKJavaThreadHoops_do6MpnKOopClosure_pnPCodeBlobClosure__v_+0x290
V  [libjvm.so+0x168b182];;  __1cHThreadsZpossibly_parallel_oops_do6FpnKOopClosure_pnPCodeBlobClosure__v_+0x96
V  [libjvm.so+0x14b2546];;  __1cKSharedHeapUprocess_strong_roots6Mbbn0AOScanningOption_pnKOopClosure_pnPCodeBlobClosure_pnQOopsInGenClosure__v_+0xc2
V  [libjvm.so+0x8e2f2a];;  __1cPG1CollectedHeapXg1_process_strong_roots6MbnKSharedHeapOScanningOption_pnKOopClosure_pnXOopsInHeapRegionClosure_6pnQOopsInGenClosure_i_v_+0x1b6
V  [libjvm.so+0x8e96ff];;  __1cJG1ParTaskEwork6Mi_v_+0x4d7
V  [libjvm.so+0x17b58b9];;  __1cKGangWorkerEloop6M_v_+0x409
V  [libjvm.so+0x17b5350];;  __1cKGangWorkerDrun6M_v_+0x2c
V  [libjvm.so+0x1341e2f];;  java_start+0x117
C  [libc.so.1+0xa7055];;  _thr_setup+0x4e
C  [libc.so.1+0xa7340];;  _lwp_start+0x0

i.e. one of the GC worker threads is scanning the stack frames of a JavaThread as part of the root-set processing during an evacuation pause. The thread that is being processed by the GC has the following stack trace:

JavaThread 0x09380000 (nid = 105) was being processed
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+0
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98

   .... ~3300 calls deep

j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.createTree(II)Lnsk/share/gc/Node;+98
j  nsk.share.gc.NonbranchyTree.<init>(IFI)V+155
j  gc.gctests.JumbleGC002.JumbleGC002$AuxiliaryThread.fillVector()V+191
j  gc.gctests.JumbleGC002.JumbleGC002$AuxiliaryThread.run()V+55
v  ~StubRoutines::call_stub