United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
JDK-8008071 : Crashed in promote_malloc_records() with Kitchensink after 19 days

Details
Type:
Bug
Submit Date:
2013-02-12
Status:
Closed
Updated Date:
2013-06-12
Project Name:
JDK
Resolved Date:
2013-02-21
Component:
hotspot
OS:
Sub-Component:
runtime
CPU:
Priority:
P2
Resolution:
Fixed
Affected Versions:
hs24
Fixed Versions:
hs24 (b34)

Related Reports
Backport:
Backport:
Backport:

Sub Tasks

Description
Got the following crash with Kitchensink after 19 days of uptime:

00000000`15a1e4e0 00000000`6e696c5a jvm!VMError::report_and_die+0x171 [c:\jdk7u2_64p\jdk7u14\hotspot\src\share\vm\utilities\vmerror.cpp @ 853]
00000000`15a1e6d0 00000000`6e99f98f jvm!topLevelExceptionFilter+0x35a [c:\jdk7u2_64p\jdk7u14\hotspot\src\os\windows\vm\os_windows.cpp @ 2555]
00000000`15a1e770 00000000`72fc1213 jvm!`java_start'::`1'::filt$0+0xf [c:\jdk7u2_64p\jdk7u14\hotspot\src\os\windows\vm\os_windows.cpp @ 422]
00000000`15a1e7a0 00000000`6e99f0a9 msvcr100!__C_specific_handler+0x97
00000000`15a1e810 00000000`771858dd jvm!__GSHandlerCheck_SEH+0x75
00000000`15a1e840 00000000`771896d7 ntdll!RtlpExecuteHandlerForException+0xd
00000000`15a1e870 00000000`77196e08 ntdll!RtlDispatchException+0x20c
00000000`15a1ef10 00000000`6e6763a1 ntdll!KiUserExceptionDispatcher+0x2e
00000000`15a1f4b0 00000000`6e676cac jvm!MemSnapshot::promote_malloc_records+0x1a1 [c:\jdk7u2_64p\jdk7u14\hotspot\src\share\vm\services\memsnapshot.cpp @ 529]
00000000`15a1f510 00000000`6e677ef1 jvm!MemSnapshot::promote+0x5c [c:\jdk7u2_64p\jdk7u14\hotspot\src\share\vm\services\memsnapshot.cpp @ 490]
00000000`15a1f580 00000000`6e69775e jvm!MemTrackWorker::run+0x101 [c:\jdk7u2_64p\jdk7u14\hotspot\src\share\vm\services\memtrackworker.cpp @ 121]
00000000`15a1f5c0 00000000`72f71db7 jvm!java_start+0x9e [c:\jdk7u2_64p\jdk7u14\hotspot\src\os\windows\vm\os_windows.cpp @ 421]

Had a debugger attached to track handle leaks so the memory dump failed. Re-running to see if I can reproduce this. 

Managed to create a callstack only dump with some information in it:

Debug session time: Tue Feb 12 17:47:27.000 2013 (UTC + 1:00)
System Uptime: not available
Process Uptime: 19 days 2:46:52.000


Local var @ 0x15a1e4a0 Type _EXCEPTION_POINTERS
   +0x000 ExceptionRecord  : 0x00000000`15a1f3e0 _EXCEPTION_RECORD
      +0x000 ExceptionCode    : 0n-1073741819
      +0x004 ExceptionFlags   : 0
      +0x008 ExceptionRecord  : (null) 
      +0x010 ExceptionAddress : 0x00000000`6e6763a1 Void
      +0x018 NumberParameters : 2
      +0x020 ExceptionInformation : [15] 0
   +0x008 ContextRecord    : 0x00000000`15a1ef10 _CONTEXT
      +0x000 P1Home           : 0x15a1f3e0
      +0x008 P2Home           : 0x20706f98
      +0x010 P3Home           : 0xfffffa60`00000000
      +0x018 P4Home           : 0x6e470000
      +0x020 P5Home           : 0
      +0x028 P6Home           : 0
      +0x030 ContextFlags     : 0x10001f
      +0x034 MxCsr            : 0x1fa0
      +0x038 SegCs            : 0x33
      +0x03a SegDs            : 0x2b
      +0x03c SegEs            : 0x2b
      +0x03e SegFs            : 0x53
      +0x040 SegGs            : 0x2b
      +0x042 SegSs            : 0x2b
      +0x044 EFlags           : 0x10246
      +0x048 Dr0              : 0
      +0x050 Dr1              : 0
      +0x058 Dr2              : 0
      +0x060 Dr3              : 0
      +0x068 Dr6              : 0
      +0x070 Dr7              : 0
      +0x078 Rax              : 0xc930
      +0x080 Rcx              : 0
      +0x088 Rdx              : 0x3edeb
      +0x090 Rbx              : 0x20706f98
      +0x098 Rsp              : 0x15a1f4b0
      +0x0a0 Rbp              : 0x15a1f4f0
      +0x0a8 Rsi              : 0x13a59ea0
      +0x0b0 Rdi              : 0xc930
      +0x0b8 R8               : 0x20510040
      +0x0c0 R9               : 0xc930
      +0x0c8 R10              : 0x139360d0
      +0x0d0 R11              : 0
      +0x0d8 R12              : 0x15a1f530
      +0x0e0 R13              : 0x13a59ea0
      +0x0e8 R14              : 0
      +0x0f0 R15              : 0
      +0x0f8 Rip              : 0x6e6763a1

                                    

Comments
Managed to reproduce after almost 3 days, crashing in promote_malloc_records:

0 00000000`1337f530 00000000`6e676cac jvm!MemSnapshot::promote_malloc_records+0x1a1 [c:\jdk7u2_64p\jdk7u14\hotspot\src\share\vm\services\memsnapshot.cpp @ 529]
01 00000000`1337f590 00000000`6e677ef1 jvm!MemSnapshot::promote+0x5c [c:\jdk7u2_64p\jdk7u14\hotspot\src\share\vm\services\memsnapshot.cpp @ 490]
02 00000000`1337f600 00000000`6e69775e jvm!MemTrackWorker::run+0x101 [c:\jdk7u2_64p\jdk7u14\hotspot\src\share\vm\services\memtrackworker.cpp @ 121]
03 00000000`1337f640 00000000`73a21db7 jvm!java_start+0x9e [c:\jdk7u2_64p\jdk7u14\hotspot\src\os\windows\vm\os_windows.cpp @ 421]
04 00000000`1337f880 00000000`73a21e53 msvcr100!_callthreadstartex+0x17 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 314]
05 00000000`1337f8b0 00000000`76d5b22d msvcr100!_threadstartex+0x7f [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 292]
06 00000000`1337f8e0 00000000`77176861 kernel32!BaseThreadInitThunk+0xd
07 00000000`1337f910 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

This time I have a complete mdmp and an hs_err file.

                                     
2013-02-18
From memSnapshot.cpp:

// an arena record can be followed by a size record, we need to remove both
       if (matched_rec->is_arena_record()) {
          MemPointerRecord* next = (MemPointerRecord*)malloc_snapshot_itr.peek_next();
          if (next->is_arena_memory_record() && next->is_memory_record_of_arena(matched_rec)) {

peek_next() seems to have returned null, Zhengyu agrees that this looks like a bug. Assigning to Zhengyu.
                                     
2013-02-18
Seems perfectly legitimate for peek_next() to return null if the matched record was the last entry.
                                     
2013-02-19
Yes, the problem here is that that promote_malloc_records() needs to handle this.


                                     
2013-02-19
ILW: HML => P2
I: H, JVM crashes
L: M
W: L, turn off NMT 
                                     
2013-02-19
URL:   http://hg.openjdk.java.net/hsx/hsx24/hotspot/rev/a398781bfe8b
User:  zgu
Date:  2013-02-21 17:01:25 +0000

                                     
2013-02-21



Hardware and Software, Engineered to Work Together