JDK-6507107 : Heapwalking causes crash on solaris-sparcv9
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 6
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: solaris
  • CPU: sparc
  • Submitted: 2006-12-21
  • Updated: 2023-08-23
  • Resolved: 2007-06-20
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 6 JDK 7 Other
6u4Fixed 7Fixed hs10Fixed
Related Reports
Relates :  
Description
Attached test using ObjectReference.referringObjects causes VM crash. I can reproduce it only on sparcv9 machines (for example vmsqe-u10-02.russia, vmsqe-u10-03.russia). Failure reproduces with mustang b105 and dolphin b04.

hs_err_pid file run through the hs_err script (hs_err_pid is attached):
#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xfecdeae4, pid=20749, tid=3
#
# Java VM: Java HotSpot(TM) Client VM (1.6.0-b105 mixed mode)
# Problematic frame:
# V  [libjvm.so+0x4deae4]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x000ab800):  VMThread [id=3]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000000;; 
;; si_signo=11	SIGSEGV
;; si_code=1	SEGV_MAPERR /* Address not mapped to object.  */

Registers:
 O0=0x0000000c O1=0xfecde630 O2=0x00000000 O3=0x00000020
 O4=0xfedfc000 O5=0x00000002 O6=0xfc0ff548 O7=0xfecdeae0
 G1=0x00000000 G2=0xfee11b34 G3=0x00000000 G4=0x00000000
 G5=0x00000000 G6=0x00000000 G7=0xfe610000 Y=0x00000000
 PC=0xfecdeae4 nPC=0xfe85bb38


Top of Stack: (sp=0xfc0ff548)
0xfc0ff548:   00000001 000ab9fc 000aba10 00000000
0xfc0ff558:   00000000 00000000 fedfc000 00000000
0xfc0ff568:   00000000 000abca0 000abc78 fffffcc8
0xfc0ff578:   fee0faac 00007000 fc0ff638 febcda14
0xfc0ff588:   f8029e50 fedfc000 fc0ff928 00000001
0xfc0ff598:   0017bcf8 ff2efa40 00000001 ff2efa54
0xfc0ff5a8:   00000010 00000011 00000011 fe8a7fe8
0xfc0ff5b8:   000349b0 fc0ff9e4 000abddc fee1287c 

Instructions: (pc=0xfecdeae4)
0xfecdead4:   7f fc d3 80 01 00 00 00 90 10 20 0c 7f ed f4 16
0xfecdeae4:   f0 06 20 00 b6 90 00 08 02 80 00 4a 92 10 20 0a 
;; 
Stack: [0xfc080000,0xfc100000),  sp=0xfc0ff548,  free space=509k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x4deae4];;  __1cOcompiledVFrameGlocals6kM_pnUStackValueCollection__+0x470
V  [libjvm.so+0x3cda1c];;  __1cUVM_HeapWalkOperationTcollect_stack_roots6MpnKJavaThread_pnUJNILocalRootsClosure__b_+0x200
V  [libjvm.so+0x3cc360];;  __1cUVM_HeapWalkOperationTcollect_stack_roots6M_b_+0xb8
V  [libjvm.so+0x3c99f8];;  __1cUVM_HeapWalkOperationEdoit6M_v_+0x238
V  [libjvm.so+0x13f82c];;  __1cMVM_OperationIevaluate6M_v_+0x88
V  [libjvm.so+0x4e5df4];;  __1cIVMThreadSevaluate_operation6MpnMVM_Operation__v_+0xd4
V  [libjvm.so+0x4e6380];;  __1cIVMThreadEloop6M_v_+0x46c
V  [libjvm.so+0xb66ac];;  __1cIVMThreadDrun6M_v_+0xa0
V  [libjvm.so+0x413a0c];;  java_start+0x120

VM_Operation (0xf3d7f89c): JVMTI VM_HeapWalkOperation, mode: safepoint, requested by thread 0x000dec00


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00030c00 JavaThread "DestroyJavaVM" [_thread_blocked, id=2]
  0x0011f800 JavaThread "Thread-2" [_thread_in_native, id=15]
  0x0011e000 JavaThread "Thread-1" [_thread_blocked, id=14]
  0x0011d800 JavaThread "Thread-0" [_thread_blocked, id=13]
  0x000eec00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=11]
  0x000ed800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=10]
  0x000e2000 JavaThread "JDWP Command Reader" daemon [_thread_in_native, id=9]
  0x000e0c00 JavaThread "JDWP Event Helper Thread" daemon [_thread_blocked, id=8]
  0x000dec00 JavaThread "JDWP Transport Listener: dt_socket" daemon [_thread_blocked, id=7]
  0x000c4800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=6]
  0x000afc00 JavaThread "Finalizer" daemon [_thread_blocked, id=5]
  0x000ae800 JavaThread "Reference Handler" daemon [_thread_blocked, id=4]

Other Threads:
=>0x000ab800 VMThread [id=3]
  0x000f0400 WatcherThread [id=12]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x0002fe28/0x0002fe58] Threads_lock - owner thread: 0x000ab800
[0x0002b6b8/0x00030308] Heap_lock - owner thread: 0x000dec00

Heap
 def new generation   total 1984K, used 1914K [0xf4000000, 0xf4220000, 0xf4710000)
  eden space 1792K, 100% used [0xf4000000, 0xf41c0000, 0xf41c0000)
  from space 192K,  63% used [0xf41f0000, 0xf420e8e8, 0xf4220000)
  to   space 192K,   0% used [0xf41c0000, 0xf41c0000, 0xf41f0000)
 tenured generation   total 4096K, used 397K [0xf4710000, 0xf4b10000, 0xf8000000)
   the space 4096K,   9% used [0xf4710000, 0xf4773628, 0xf4773800, 0xf4b10000)
 compacting perm gen  total 12288K, used 2348K [0xf8000000, 0xf8c00000, 0xfc000000)
   the space 12288K,  19% used [0xf8000000, 0xf824b280, 0xf824b400, 0xf8c00000)
No shared spaces configured.

Dynamic libraries:
0x00010000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/bin/java
0xff3f8000 	/lib/libthread.so.1
0xff380000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/bin/../lib/sparc/jli/libjli.so
0xff3a0000 	/lib/libdl.so.1
0xff200000 	/lib/libc.so.1
0xff358000 	/platform/SUNW,Ultra-5_10/lib/libc_psr.so.1
0xfe800000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/client/libjvm.so
0xff320000 	/lib/libsocket.so.1
0xff350000 	/usr/lib/libsched.so.1
0xff1e0000 	/lib/libm.so.1
0xff1b0000 	/usr/lib/libCrun.so.1
0xff190000 	/lib/libdoor.so.1
0xff080000 	/lib/libnsl.so.1
0xfef80000 	/lib/libm.so.2
0xff160000 	/lib/libscf.so.1
0xff140000 	/lib/libuutil.so.1
0xff050000 	/lib/libmd5.so.1
0xfef60000 	/platform/SUNW,Ultra-5_10/lib/libmd5_psr.so.1
0xfef40000 	/lib/libmp.so.2
0xfeeb0000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/native_threads/libhpi.so
0xfee50000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/libjdwp.so
0xfe7d0000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/libnpt.so
0xfe7b0000 	/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3
0xfe790000 	/usr/lib/iconv/UTF-8%8859-1.so
0xfe6e0000 	/usr/lib/iconv/8859-1%UTF-8.so
0xfe6a0000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/libverify.so
0xfe660000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/libjava.so
0xfe630000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/libzip.so
0xfe560000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/libdt_socket.so
0xfe510000 	/lib/nss_nis.so.1
0xfc320000 	/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/libnet.so

VM Arguments:
jvm_args: -Xdebug -Xrunjdwp:transport=dt_socket,address=vmsqe-u10-02:50713,suspend=y
java_command: HeapwalkingDebuggee
Launcher Type: SUN_STANDARD

Environment Variables:
CLASSPATH=/set/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/lib/tools.jar:/home/sb196040/eclipse_jpda/JDIMultistress/bin
PATH=/usr/dt/bin:/usr/openwin/bin:/usr/ccs/bin:/usr/bin:/usr/sbin:/sbin:/usr/ucb:/usr/dist/local/exe:/usr/sfw/bin:/usr/dist/exe:/usr/lib/lp/postscript:.
LD_LIBRARY_PATH=/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc/client:/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/lib/sparc:/net/vmsqe-amd-01.russia/export3/vmsqe/jdk-builds/JDK6b105/solaris-sparcv9/jre/../lib/sparc
SHELL=/bin/bash
DISPLAY=d-espb04-127-83:0.0

Signal Handlers:
SIGSEGV: [libjvm.so+0x4e4300], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0x4e4300], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0x172aa8], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x172aa8], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x172aa8], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGHUP: [libjvm.so+0x4155c0], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGINT: [libjvm.so+0x4155c0], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0x4155c0], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x4155c0], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0x4155c0], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x4155c0], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIG39: [libjvm.so+0x4180e8], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIG40: [libjvm.so+0x172aa8], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c


---------------  S Y S T E M  ---------------

OS:                         Solaris 10 3/05 s10_74L2a SPARC
           Copyright 2005 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                            Assembled 22 January 2005

uname:SunOS 5.10 Generic_118822-30 sun4u  (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:0.15 0.15 0.12

CPU:total 1 has_v8, has_v9, has_vis1

Memory: 8k page, physical 524288k(295120k free)

vm_info: Java HotSpot(TM) Client VM (1.6.0-b105) for solaris-sparc, built on Nov 29 2006 01:43:57 by "" with unknown Workshop:0x580
Regression test 
	closed/compiler/6507107/HeapwalkingTest.java
failed during PIT for b13 because of crash in debuggee VM with hs_err_pid similar to hs_err_pid from first description (full hs_err_pid is attached):
#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xfeb56fcb, pid=9230, tid=3
#
# Java VM: Java HotSpot(TM) Client VM (20070517174904.jrose.c2_to_main_baseline mixed mode solaris-x86)
# Problematic frame:
# V  [libjvm.so+0x356fcb]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x080ed400):  VMThread [stack: 0xf41be000,0xf41fe000] [id=3]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00000000;;

Registers:
EAX=0xfeb56fcb, EBX=0xfec44000, ECX=0x00000000, EDX=0x00000000
ESP=0xf41fd844, EBP=0xf41fd880, ESI=0x00000000, EDI=0x02000000
EIP=0xfeb56fcb, EFLAGS=0x00010283

Top of Stack: (sp=0xf41fd844)
0xf41fd844:   00000000 00000000 fec44000 00000000
0xf41fd854:   00000019 080ed6d0 feb58019 080ed820
0xf41fd864:   00000004 080ed820 fec44000 feaf7773
0xf41fd874:   080ed7b0 00000171 f41fd914 f41fd914
0xf41fd884:   feb567b5 080ed6d0 080ed808 fec5ed68
0xf41fd894:   080ed6d0 fec44000 f41fd914 080edae4
0xf41fd8a4:   080edae4 f41fd8e8 fe9e834f 00000000
0xf41fd8b4:   fec5ed68 f8269810 fec44000 f41fdad0

Instructions: (pc=0xfeb56fcb)
0xfeb56fbb:   00 00 50 e8 21 f3 e6 ff 83 c4 08 e8 f1 5a f7 ff
0xfeb56fcb:   8b 06 89 45 ec 6a 0c e8 75 f0 cf ff 83 c4 04 85
;;
Stack: [0xf41be000,0xf41fe000],  sp=0xf41fd844,  free space=254k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x356fcb];;  __1cOcompiledVFrameScreate_stack_value6kMpnKScopeValue__pnKStackValue__+0x133
V  [libjvm.so+0x3567b5];;  __1cOcompiledVFrameGlocals6kM_pnUStackValueCollection__+0x109
V  [libjvm.so+0x2a215b];;  __1cUVM_HeapWalkOperationTcollect_stack_roots6MpnKJavaThread_pnUJNILocalRootsClosure__b_+0x26f
V  [libjvm.so+0x2a11e2];;  __1cUVM_HeapWalkOperationTcollect_stack_roots6M_b_+0x72
V  [libjvm.so+0x29fdcc];;  __1cUVM_HeapWalkOperationEdoit6M_v_+0xd8
V  [libjvm.so+0xec816];;  __1cMVM_OperationIevaluate6M_v_+0x76
V  [libjvm.so+0xec6e8];;  __1cIVMThreadSevaluate_operation6MpnMVM_Operation__v_+0xac
V  [libjvm.so+0xa6210];;  __1cIVMThreadEloop6M_v_+0x268
V  [libjvm.so+0xa5eb8];;  __1cIVMThreadDrun6M_v_+0x88
V  [libjvm.so+0x2ccbae];;  java_start+0xee
C  [libc.so.1+0x9fd36];;  _thr_setup+0x4e
C  [libc.so.1+0xa0020];;  _lwp_start+0x0

VM_Operation (0xf40dd9a0): JVMTI VM_HeapWalkOperation, mode: safepoint, requested by thread 0x0811f800


To reproduce:
	ssh vm-v20z-3.sfbay
	cd /net/gtee.sfbay/export/misc/6507107/
	/net/gtee.sfbay/export/bin/run_jtreg -suite HS_REGRESSION -jdk /net/gtee.sfbay/export/gtee2.0/results/JDK7/NIGHTLY/VM/JDK/baseline -test closed/compiler/6507107/HeapwalkingTest.java -arch solaris-i586
(script /net/gtee.sfbay/export/bin/run_jtreg just runs jtreg with specified JDK)
Regression test 
	closed/compiler/6507107/HeapwalkingTest.java
failed with crash on solaris-amd64 during PIT for b14 (crash happened in the same place, hs_err_pid in /net/gtee.sfbay/export/misc/6507107/JTwork/closed/compiler/6507107/HeapwalkingTest).

To reproduce:
	ssh vm-x86-1.sfbay
	cd /net/gtee.sfbay/export/misc/6507107/
	/net/gtee.sfbay/export/bin/run_jtreg -suite HS_REGRESSION -jdk /net/gtee.sfbay/export/gtee2.0/results/JDK7/NIGHTLY/VM/JDK/baseline -test closed/compiler/6507107/HeapwalkingTest.java -arch solaris-amd64

Comments
EVALUATION I think your JDK setup is incorrect. When I run it against the bits in our workspace it all works fine. Running against your amd64 bits I get this: /net/gtee.sfbay/export/gtee2.0/results/JDK7/NIGHTLY/VM/JDK/baseline/solaris-amd64/bin/java -classpath .:$j160/lib/tools.jar HeapwalkingTest debuggee out>>>Thread[Thread-0,5,main] started debuggee out>>>Thread[Thread-1,5,main] started Start test Iteration: 1 debuggee out>>># debuggee out>>># An unexpected error has been detected by Java Runtime Environment: debuggee out>>># debuggee out>>># SIGSEGV (0xb) at pc=0xfeb57017, pid=24572, tid=3 debuggee out>>># debuggee out>>># Java VM: Java HotSpot(TM) Client VM (1.7.0-ea-b13 mixed mode solaris-x86) debuggee out>>># Problematic frame: debuggee out>>># V [libjvm.so+0x357017] debuggee out>>># debuggee out>>># An error report file with more information is saved as: debuggee out>>># /net/smite/export/ws/sux/test/closed/compiler/6507107/hs_err_pid24572.log debuggee out>>># debuggee out>>># If you would like to submit a bug report, please visit: debuggee out>>># http://java.sun.com/webapps/bugreport/crash.jsp debuggee out>>># Exception in thread "main" com.sun.jdi.VMDisconnectedException Notice that the JVM version is reported as b13 instead of 20070614223257.steved.c2_to_main_baseline. I don't think your jdk for solaris-amd64 has the proper versions of libjvm for the 32 bit vm. hsdev-5 closed/compiler/6507107 % /net/gtee.sfbay/export/gtee2.0/results/JDK7/NIGHTLY/VM/JDK/baseline/solaris-amd64/bin/java -client -version java version "1.7.0-ea" Java(TM) SE Runtime Environment (build 1.7.0-ea-b13) Java HotSpot(TM) Client VM (build 1.7.0-ea-b13, mixed mode) hsdev-5 closed/compiler/6507107 % /net/gtee.sfbay/export/gtee2.0/results/JDK7/NIGHTLY/VM/JDK/baseline/solaris-i586/bin/java -client -version java version "1.7.0-ea" Java(TM) SE Runtime Environment (build 1.7.0-ea-b13) Java HotSpot(TM) Client VM (build 20070614223257.steved.c2_to_main_baseline, mixed mode) By the way, all the directories you pointed me at have the permission set to drwx--x--x which doesn't allow me to examine them. You should really fix your umask so this isn't a problem or maybe the jtreg harness should do this.
18-06-2007

SUGGESTED FIX Webrev: http://prt-web.sfbay.sun.com/net/prt-archiver.sfbay/data/archived_workspaces/main/c2_baseline/2007/20070606093209.never.6507107/workspace/webrevs/webrev-2007.06.06/index.html Fixed 6507107: Heapwalking causes crash on solaris-sparcv9 The previous fix for this was incomplete since it was still broken on x86. In some of our stubs we tried to be smart about not saving off kept on the FPU stack in deoptable form if we weren't in the deopt blob but this means that looking at the value of locals without deopting can SEGV if you look at locals which are kept on the FPU stack. http://javaweb.sfbay/~never/webrev/6507107 Reviewed by: kvn, sgoldman Fix verified (y/n): y test case on sparc and intel with UseSSE=0,1,2
07-06-2007

EVALUATION I thought I tested on intel but apparently not... The slow paths stubs on intel don't always describe the location of fpu values even when they save this. This was motivated by the extra cost of saving the FPU registers in a deoptable form when we aren't using SSE. We really need to be able to find the registers all time so I've fixed it to save and describe them all the time as we'd expect.
24-05-2007

SUGGESTED FIX Original workspace: smite:/export/ws/6507107 Parent workspace: /net/jano.sfbay/export/disk05/hotspot/ws/main/c2_baseline Submitter: never PRT data: /net/prt-web.sfbay/prt-workspaces/20070418160846.never.6507107 Archived data: /net/prt-archiver.sfbay/data/archived_workspaces/main/c2_baseline/2007/20070418160846.never.6507107/ Webrev: http://prt-web.sfbay.sun.com/net/prt-archiver.sfbay/data/archived_workspaces/main/c2_baseline/2007/20070418160846.never.6507107/workspace/webrevs/webrev-2007.04.18/index.html Fixed 6507107: Heapwalking causes crash on solaris-sparcv9 Some refactoring of the code generation of stubs in c1_Runtime1_sparc.cpp causes us to lose track of the callee saved oopmaps on some stubs. Deopt still works correctly because we are properly saving and restoring registers and the deopt blob properly describes their location but if you try to traverse debug info when you aren't deoptimizing you may not be able to find floating point registers on sparc. The fix is simply to properly return the OopMapSet so that it's recorded with the blob. I also added code to make sure that blobs which need them have them which exposed that the register_finalizer helper was also missing it's callee saved oopmap. http://javaweb.sfbay/~never/webrev/6507107 Reviewed by: nips, kvn Fix verified (y/n): y test case
19-04-2007

EVALUATION Some refactoring of the code generation of stubs in c1_Runtime1_sparc.cpp cause us to lose the setting of the callee saved oopmaps on some stubs. Deopt still works correctly because we are properly saving and restoring registers and the deopt blob properly describes their location but if you try to traverse debug info when you aren't deoptimizing you may node be able to find floating point values. The fix is simply to properly return the OopMapSet so that it's recorded with the blob.
21-03-2007