JDK-8144947 : Crash with memory stomping error Trailer guard is BROKEN
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 9
  • Priority: P2
  • Status: Closed
  • Resolution: Duplicate
  • Submitted: 2015-12-08
  • Updated: 2016-01-19
  • Resolved: 2016-01-19
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 9
9Resolved
Related Reports
Duplicate :  
JDK-8147477 - com/sun/management/HotSpotDiagnosticMXBean/CheckOrigin.java is failing for the jdk9/hs snapshot control job
Relates :  
JDK-8146222 - assert(_initialized) failed: TLS not initialized yet!
Relates :  
JDK-8147604 - CheckOrigin.java test needs specific exit code for exceptions (versus crashes)
Relates :  
JDK-8147477 - com/sun/management/HotSpotDiagnosticMXBean/CheckOrigin.java is failing for the jdk9/hs snapshot control job
Relates :  
JDK-8147481 - VM crash fatal error: corrupted C heap (runtime.threads.Threads2a.Threads2a)
Description
Test causes VM to overwrite memory

## nof_mallocs = 38979, nof_frees = 6359
## memory stomp:
GuardedMemory(0x3352fb04) base_addr=0x342b29a0 tag=0x00000000 user_size=32764 user_data=0x342b29b8
  Header guard @0x342b29a0 is OK
  Trailer guard @0x342ba9b4 is BROKEN
  User data appears to be in use
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=\\os.cpp:517
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (C:\\jprt\\T\\P1\\214445.cplummer\\s\\hotspot\\src\\share\\vm\\runtime\\os.cpp:517), pid=114424, tid=147020
#  fatal error: memory stomping error
Comments
We're closing this as a duplicate of JDK-8147477, this bug is tracking the memory stomp as a result of the CheckOrigins.java failure. The earlier reported memory stomps will have to be tracked by a new bug once the CheckOrigins issue has been resolved.
19-01-2016
Hi, this is not a compiler issue. Here are the reasons why not. I reproduced the memory stomping failure using information from the GC nightly run of 2015-12-23 1214044.JAVASE.NIGHTLY.VM.GC_Baseline-G1.2015-12-23 I used the following: - test com/sun/management/HotSpotDiagnosticMXBean/CheckOrigin.java - build 2015-12-23-232608.cphillim.jdk9-sponsor-rp - machine: the same linux-x86_32 machine as reported in the failure. In the default configuration, the test fails with the following output: ./cmd.sh Command line: [/tmp/zmajo/8144947/testing-checkorigin/../jdk.232608.cphillim/bin/java -XX:+UseConcMarkSweepGC -XX:+UseCodeAging -XX:+Use CerealGC -XX:Flags=/tmp/CheckOriginFlags8374274195822851433.tmp -cp ./JTwork/classes/com/sun/management/HotSpotDiagnosticMXBean/:./JTwok k/classes/lib/testlibrary CheckOrigin -runtests ] VM option '+PrintSafepointStatistics' VM option '+IgnoreUnrecognizedVMOptions' VM option '+PrintVMOptions' VM option '+UseGOneGC' VM option '+UseConcMarkSweepGC' VM option '+UseCodeAging' VM option '+UseCerealGC' VM option 'Flags=/tmp/CheckOriginFlags8374274195822851433.tmp' VM option '+TraceExceptions' VM option '+UseCMSGC' ## nof_mallocs = 669, nof_frees = 54 ## memory stomp: GuardedMemory(0xf776a038) base_addr=0xf7769e28 tag=0x422d39fc user_size=4151746276 user_data=0xf7769e40 Header guard @0xf7769e28 is BROKEN sub process exit == 139 Exception in thread "main" java.lang.Exception: Unexpected exit code from subprocess == 139 The test works the following way: after launched for the first time, the test launches a new copy of itself. I modified the test so that it launches the copy of itself with compilation disabled: diff --git a/test/com/sun/management/HotSpotDiagnosticMXBean/CheckOrigin.java b/test/com/sun/management/HotSpotDiagnosticMXBean/CheckOri\ gin.java --- a/test/com/sun/management/HotSpotDiagnosticMXBean/CheckOrigin.java +++ b/test/com/sun/management/HotSpotDiagnosticMXBean/CheckOrigin.java @@ -1,3 +1,4 @@ + /* * Copyright (c) 2013, 2015, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. @@ -61,6 +62,9 @@ ProcessBuilder pb = ProcessTools. createJavaProcessBuilder( + "-XX:-UseCompiler", + "-Xint", + "-XX:-RenumberLiveNodes", "-XX:+UseConcMarkSweepGC", // this will cause UseParNewGC to be FLAG_SET_ERGO "-XX:+UseCodeAging", "-XX:+UseCerealGC", // Should be ignored. Also, I launch the test with compilation disabled (i.e., with only using the interpreter). Here is the output: bash-4.1$ ./cmd.sh Command line: [/tmp/zmajo/8144947/testing-checkorigin/../jdk.232608.cphillim/bin/java -XX:-UseCompiler -Xint -XX:-RenumberLiveNodes -XX:\ +UseConcMarkSweepGC -XX:+UseCodeAging -XX:+UseCerealGC -XX:Flags=/tmp/CheckOriginFlags4132298253791109250.tmp -cp ./JTwork/classes/com/s\ un/management/HotSpotDiagnosticMXBean/:./JTwork/classes/lib/testlibrary CheckOrigin -runtests ] VM option '+PrintSafepointStatistics' VM option '+IgnoreUnrecognizedVMOptions' VM option '+PrintVMOptions' VM option '+UseGOneGC' VM option '-UseCompiler' VM option '-RenumberLiveNodes' VM option '+UseConcMarkSweepGC' VM option '+UseCodeAging' VM option '+UseCerealGC' VM option 'Flags=/tmp/CheckOriginFlags4132298253791109250.tmp' VM option '+TraceExceptions' VM option '+UseCMSGC' ## nof_mallocs = 676, nof_frees = 58 ## memory stomp: GuardedMemory(0xf775a038) base_addr=0xf7759e28 tag=0x422d39fc user_size=4151680740 user_data=0xf7759e40 Header guard @0xf7759e28 is BROKEN sub process exit == 139 Exception in thread "main" java.lang.Exception: Unexpected exit code from subprocess == 139 at CheckOrigin.main(CheckOrigin.java:89) That is, the memory stomping error is still there, even though compilation has been disabled. Therefore, this is not a compiler issue and I'm reassigning the issue to the runtime team for further investigation. I have set up the reproducer, please contact me if you need access to it. Thank you and best regards, Zoltan
08-01-2016
Thanks Christian! I will file a new bug for the TLS issue.
27-12-2015
00 0261eeac 6d9226d7 jvm!VMError::report_and_die+0x870 [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\utilities\vmerror.cpp @ 1304] 01 0261eee0 6d605ae8 jvm!VMError::report_and_die+0x27 [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\utilities\vmerror.cpp @ 1083] 02 0261ef00 6d8fb03f jvm!report_vm_error+0x48 [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\utilities\debug.cpp @ 218] 03 0261ef14 6d8477de jvm!ThreadLocalStorage::thread+0x1f [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\os\windows\vm\threadlocalstorage_windows.cpp @ 52] 04 (Inline) -------- jvm!Thread::current_or_null_safe+0x5 05 0261f06c 6d724e78 jvm!topLevelExceptionFilter+0x3e [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\os\windows\vm\os_windows.cpp @ 2357] 06 0261f074 6dcb2aa5 jvm!JNI_CreateJavaVM+0x78 [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\prims\jni.cpp @ 4107] 07 0261f088 6dcb2b33 msvcr120!_EH4_CallFilterFunc+0x12 [f:\dd\vctools\crt\crtw32\misc\i386\exsup4.asm @ 391] 08 0261f0b4 6d947682 msvcr120!_except_handler4_common+0x8e [f:\dd\vctools\crt\crtw32\misc\i386\chandler4.c @ 360] 09 0261f0d4 76fd00b1 jvm!_except_handler4+0x1e [f:\dd\vctools\crt\crtw32\misc\i386\chandler4gs.c @ 91] 0a 0261f0f8 76fd0083 ntdll!ExecuteHandler2+0x26 0b 0261f1c4 76fd07ff ntdll!ExecuteHandler+0x24 0c 0261f1c4 6d698956 ntdll!KiUserExceptionDispatcher+0xf 0d (Inline) -------- jvm!GuardedMemory::Guard::verify+0x9 0e 0261f520 6d83dff8 jvm!GuardedMemory::print_on+0x86 [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\memory\guardedmemory.cpp @ 67] 0f 0261f534 6d83b1bb jvm!verify_memory+0xb8 [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\runtime\os.cpp @ 529] 10 0261f574 6d48bb5b jvm!os::free+0x7b [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\runtime\os.cpp @ 692] 11 0261f584 6d48eb49 jvm!ScopedVMInitArgs::~ScopedVMInitArgs+0x2b [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\runtime\arguments.cpp @ 3551] 12 0261f5dc 6d8f2f24 jvm!Arguments::parse+0x279 [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\runtime\arguments.cpp @ 4125] 13 0261f6b0 6d722a6f jvm!Threads::create_vm+0x94 [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\runtime\thread.cpp @ 3418] 14 0261f714 6d724e4f jvm!JNI_CreateJavaVM_inner+0x6f [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\prims\jni.cpp @ 4016] 15 0261f754 00282551 jvm!JNI_CreateJavaVM+0x4f [c:\jprt\t\p1\232608.cphillim\s\hotspot\src\share\vm\prims\jni.cpp @ 4105] WARNING: Stack unwind information not available. Following frames may be wrong.
27-12-2015
The: # assert(_initialized) failed: TLS not initialized yet! is a second problem here. Even if the GuardedMemory problem is triggering this there should not be a code path that hits the TLS assert. Unfortunately the hs_err log shows nothing helpful: V [jvm.dll+0x185ae8] report_vm_error+0x48;; ?report_vm_error@@YAXPBDH00ZZ+0x48 V [jvm.dll+0x47b03f] ThreadLocalStorage::thread+0x1f;; ?thread@ThreadLocalStorage@@SAPAVThread@@XZ+0x1f V [jvm.dll+0x2a4e78] JNI_CreateJavaVM+0x78;; _JNI_CreateJavaVM@12+0x78 Need to know exactly where in CreateJavaVM we called ThreadLocalStorage::thread.
27-12-2015
Giving this to the compiler team
09-12-2015
Looked at the dump from the compiler nightly last week: GuardedMemory(0x3798f6f8) base_addr=0x35ba4620 tag=0x00000000 user_size=32764 user_data=0x35ba4638 Header guard @0x35ba4620 is OK Trailer guard @0x35bac634 is BROKEN 0:000> db 35bac634-0x4 l8 35bac630 ab ab ab ab 00 ab ab ab 0:000> s 0x0 L?0xFFFFFFFF 34 c6 ba 35 36ddd510 34 c6 ba 35 c4 03 01 00-00 d0 83 35 40 fa dd 36 4..5.......5@..6 0:000> !address 36ddd510 Mapping file section regions... Mapping module regions... Mapping PEB regions... Mapping TEB and stack regions... Mapping heap regions... Mapping page heap regions... Mapping other regions... Mapping stack trace database regions... Mapping activation context regions... Usage: Stack Base Address: 36ddd000 End Address: 36de0000 Region Size: 00003000 ( 12.000 kB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 36ce0000 Allocation Protect: 00000004 PAGE_READWRITE More info: ~68k 0:000> ~68k # ChildEBP RetAddr 00 36ddfb08 76152cc7 ntdll!NtWaitForSingleObject+0xc 01 36ddfb7c 76152c02 KERNELBASE!WaitForSingleObjectEx+0x99 02 36ddfb90 71313b9d KERNELBASE!WaitForSingleObject+0x12 03 36ddfbc0 7125c21a jvm!os::PlatformEvent::park+0x10d [c:\jprt\t\p1\214318.mikael\s\hotspot\src\os\windows\vm\os_windows.cpp @ 5080] 04 (Inline) -------- jvm!ParkCommon+0x4a [c:\jprt\t\p1\214318.mikael\s\hotspot\src\share\vm\runtime\mutex.cpp @ 423] 05 36ddfbe0 7125d927 jvm!Monitor::IWait+0x10a [c:\jprt\t\p1\214318.mikael\s\hotspot\src\share\vm\runtime\mutex.cpp @ 799] 06 36ddfc04 710f7f1d jvm!Monitor::wait+0x287 [c:\jprt\t\p1\214318.mikael\s\hotspot\src\share\vm\runtime\mutex.cpp @ 1136] 07 36ddfc68 710f7c50 jvm!CompileQueue::get+0x8d [c:\jprt\t\p1\214318.mikael\s\hotspot\src\share\vm\compiler\compilebroker.cpp @ 367] 08 36ddfcd0 71291e38 jvm!CompileBroker::compiler_thread_loop+0x140 [c:\jprt\t\p1\214318.mikael\s\hotspot\src\share\vm\compiler\compilebroker.cpp @ 1529] 09 36ddfd24 71291344 jvm!JavaThread::thread_main_inner+0xd8 [c:\jprt\t\p1\214318.mikael\s\hotspot\src\share\vm\runtime\thread.cpp @ 1718] 0a 36ddfd58 713126b1 jvm!JavaThread::run+0x184 [c:\jprt\t\p1\214318.mikael\s\hotspot\src\share\vm\runtime\thread.cpp @ 1701] 0b 36ddfe94 7251c01d jvm!java_start+0xd1 [c:\jprt\t\p1\214318.mikael\s\hotspot\src\os\windows\vm\os_windows.cpp @ 443] 0c 36ddfecc 7251c001 msvcr120!_callthreadstartex+0x1b [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 376] 0d 36ddfed8 759e7c04 msvcr120!_threadstartex+0x7c [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 354] 0e 36ddfeec 77b3ad1f kernel32!BaseThreadInitThunk+0x24 0f 36ddff34 77b3acea ntdll!__RtlUserThreadStart+0x2f 10 36ddff44 00000000 ntdll!_RtlUserThreadStart+0x1b
09-12-2015
## memory stomp: GuardedMemory(0x3352fb04) base_addr=0x342b29a0 tag=0x00000000 user_size=32764 user_data=0x342b29b8 Header guard @0x342b29a0 is OK Trailer guard @0x342ba9b4 is BROKEN 0:013> db 0x342ba9b4-0x4 L8 342ba9b0 ab ab ab ab 00 ab ab ab Someone has written 00 to our tail guard at address 0x342ba9b4 Searching the memory for references to 0x342ba9b4 0:013> s 0x0 L?0xFFFFFFFF b4 a9 2b 34 32afd178 b4 a9 2b 34 c4 03 01 00-00 84 f3 31 a8 f6 af 32 ..+4.......1...2 32afeaa0 b4 a9 2b 34 c4 03 01 00-00 84 f3 31 a8 f6 af 32 ..+4.......1...2 0:013> dd 32afd178 l1 32afd178 342ba9b4 0:013> dd 32afeaa0 l1 32afeaa0 342ba9b4 0:013> !address 32afd178 Usage: Stack Base Address: 32afd000 End Address: 32b00000 Region Size: 00003000 ( 12.000 kB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 32a00000 Allocation Protect: 00000004 PAGE_READWRITE More info: ~13k 0:013> !address 32afeaa0 Usage: Stack Base Address: 32afd000 End Address: 32b00000 Region Size: 00003000 ( 12.000 kB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 32a00000 Allocation Protect: 00000004 PAGE_READWRITE More info: ~13k Both are on the stack of thread ~13: 0:013> ~13k # ChildEBP RetAddr 00 32aff770 756a2cc7 ntdll!NtWaitForSingleObject+0xc 01 32aff7e4 756a2c02 KERNELBASE!WaitForSingleObjectEx+0x99 02 32aff7f8 6ea6757d KERNELBASE!WaitForSingleObject+0x12 03 32aff828 6e9af89a jvm!os::PlatformEvent::park+0x10d [c:\jprt\t\p1\214445.cplummer\s\hotspot\src\os\windows\vm\os_windows.cpp @ 5072] 04 (Inline) -------- jvm!ParkCommon+0x4a [c:\jprt\t\p1\214445.cplummer\s\hotspot\src\share\vm\runtime\mutex.cpp @ 423] 05 32aff848 6e9b0fee jvm!Monitor::IWait+0x10a [c:\jprt\t\p1\214445.cplummer\s\hotspot\src\share\vm\runtime\mutex.cpp @ 799] 06 32aff86c 6e849c3d jvm!Monitor::wait+0x28e [c:\jprt\t\p1\214445.cplummer\s\hotspot\src\share\vm\runtime\mutex.cpp @ 1136] 07 32aff8d0 6e8498f3 jvm!CompileQueue::get+0x8d [c:\jprt\t\p1\214445.cplummer\s\hotspot\src\share\vm\compiler\compilebroker.cpp @ 367] 08 32aff940 6e9e55bc jvm!CompileBroker::compiler_thread_loop+0x173 [c:\jprt\t\p1\214445.cplummer\s\hotspot\src\share\vm\compiler\compilebroker.cpp @ 1529] 09 32aff994 6e9e4a59 jvm!JavaThread::thread_main_inner+0xfc [c:\jprt\t\p1\214445.cplummer\s\hotspot\src\share\vm\runtime\thread.cpp @ 1711] 0a 32aff9c8 6ea660d8 jvm!JavaThread::run+0x199 [c:\jprt\t\p1\214445.cplummer\s\hotspot\src\share\vm\runtime\thread.cpp @ 1694] 0b 32affb84 707fc01d jvm!java_start+0xd8 [c:\jprt\t\p1\214445.cplummer\s\hotspot\src\os\windows\vm\os_windows.cpp @ 445] 0c 32affbbc 707fc001 msvcr120!_callthreadstartex+0x1b [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 376] 0d 32affbc8 77027c04 msvcr120!_threadstartex+0x7c [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 354] 0e 32affbdc 77e1ad1f kernel32!BaseThreadInitThunk+0x24 0f 32affc24 77e1acea ntdll!__RtlUserThreadStart+0x2f 10 32affc34 00000000 ntdll!_RtlUserThreadStart+0x1b
09-12-2015