JDK-8029106 : JVM crashes in Metachunk::Metachunk during parallel class redefinition (PrivateMLetController, anonymous-simple_copy_1)
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: hs25
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2013-11-25
  • Updated: 2014-01-14
  • Resolved: 2013-12-11
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 8 JDK 9 Other
8Fixed 9Fixed hs25Fixed
Related Reports
Relates :  
Description
Hard to reproduce crash when running parallel class loading and redefinition tests (/bin/classes/runtime/ParallelClassLoading/shared/hierarchies/static-load/dynamic-init/anonymous/simple, class runtime.ParallelClassLoading.shared.PrivateMLetController)

Comments
Release team: Approved for fixing
11-12-2013

8-critical-request: We risk crashing whenever Metaspace is allocated near the end of the virtual space of a machine. I have a one-liner fix (plus unit tests) for this, which is out for review. Review request: http://mail.openjdk.java.net/pipermail/hotspot-dev/2013-December/011947.html
09-12-2013

A. I don't know how you reproduced it and I couldn't. B. I'm mad that I couldn't debug the core file on the Aurora machine, C. Do we need more overflow protection in the metaspace code? and D. CMS was a red herring (ie. wrong direction) Thank you for figuring this out.
06-12-2013

It looks like we have a overflow bug in VirtualSpaceNode::is_available(). See: (gdb) p *this->_current_virtual_space $4 = { CHeapObj = {<No data fields>}, _next = 0x0, _reserved = { _start = 0xffef7000, _word_size = 262144 }, _rs = { _base = 0xffef7000 "", _size = 1048576, _noaccess_prefix = 0, _alignment = 4096, _special = False, _executable = False }, _virtual_space = { _low_boundary = 0xffef7000 "", _high_boundary = 0xffff7000 <Address 0xffff7000 out of bounds>, _low = 0xffef7000 "", _high = 0xffff7000 <Address 0xffff7000 out of bounds>, _special = False, _executable = False, _lower_high = 0xffef7000 "", _middle_high = 0xffff7000 <Address 0xffff7000 out of bounds>, _upper_high = 0xffff7000 <Address 0xffff7000 out of bounds>, _lower_high_boundary = 0xffef7000 "", _middle_high_boundary = 0xffff7000 <Address 0xffff7000 out of bounds>, _upper_high_boundary = 0xffff7000 <Address 0xffff7000 out of bounds>, _lower_alignment = 4096, _middle_alignment = 4096, _upper_alignment = 4096 }, _top = 0x2000, _container_count = 26 } The VirtualSpaceNode spans [0xffef7000, 0xffff7000) and we crash when we try to allocate something at 0xffff7000. But, the _top pointer is not within that range, its 0x2000 ! Here's a simplified version of take_from_committed: Metachunk* VirtualSpaceNode::take_from_committed(size_t chunk_word_size) { MetaWord* chunk_limit = top(); if (!is_available(chunk_word_size)) { return NULL; } // Take the space (bump top on the current virtual space). inc_top(chunk_word_size); // Initialize the chunk Metachunk* result = ::new (chunk_limit) Metachunk(chunk_word_size, this); return result; } I think the bug is that is_available doesn't handle overflows and we incorrectly say that it's enough space to allocate the current committed area: bool is_available(size_t word_size) { return _top + word_size <= end(); } inc_top changed our _top value. Calculating backwards give: _top - word_size => p /x (size_t)0x2000 - 11264 * 4 => 0xffff7000 So, the likely scenario is that _top was 0xffff7000 when we entered the take_from_committed and _top + word_sized overflowed and gave 0x2000. Since this is less than end() is_available() returns true, and we try to allocate outside the memory area.
06-12-2013

I think this has something to do with CMS class unloading. Have we seen these metaspace failures without CMS?
06-12-2013

This one is trying to allocate a Metachunk from the last region of allocated memory. I don't see how this is possible from the code. siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0xffff6000;; V [libjvm.so+0x683029] Metachunk::Metachunk(unsigned int, VirtualSpaceNode*)+0x9;; Metachunk::Metachunk(unsigned int, VirtualSpaceNode*)+0x9 pmap: ... ffed5000-ffef6000 rw-p 00000000 00:00 0 [stack] ffef6000-ffff6000 rw-p 00000000 00:00 0 Unless the mmap region starting at 0xffff6000 was unmapped (when /tmp is filled?)
06-12-2013

ILW: HLM => P3 Impact: High (crash) Likelihood: Low (hard to reproduce) Workaround: Medium (a corner case, running parallel class redefinition with JFR ON)
25-11-2013

# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0xf6fb1029, pid=3216, tid=4151520112 # # JRE version: Java(TM) SE Runtime Environment (8.0-b116) (build 1.8.0-ea-b116) # Java VM: Java HotSpot(TM) Server VM (25.0-b58 mixed mode linux-x86 ) # Problematic frame: # V [libjvm.so+0x683029] Metachunk::Metachunk(unsigned int, VirtualSpaceNode*)+0x9 # # Core dump written. Default location: /export/local/aurora/sandbox/results/ResultDir/anonymous-simple_copy_1/core or core.3216 # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0xf6807c00): JavaThread "Main Thread" [_thread_in_vm, id=3220, stack(0xf76e2000,0xf7733000)] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0xffff6000;; Registers: EAX=0x00003300, EBX=0x9d5f8b30, ECX=0x000032aa, EDX=0xffff6000 ESP=0xf77313d8, EBP=0xf77313d8, ESI=0xffff6000, EDI=0x00003300 EIP=0xf6fb1029, EFLAGS=0x00010286, CR2=0xffff6000 Top of Stack: (sp=0xf77313d8) 0xf77313d8: f7731408 f6fb5e18 ffff6000 00003300 0xf77313e8: 9d5f8b30 ffff6000 0000150c 00008000 0xf77313f8: a1c66708 00000000 00003300 00000001 0xf7731408: f7731438 f6fb75f4 a1c66708 000032aa 0xf7731418: 00003300 00008000 9e00dc48 0000000c 0xf7731428: 0934ad50 9e00e098 000032aa 00000000 0xf7731438: f7731468 f6fb77d1 9e00e098 000032aa 0xf7731448: 00000000 00000000 9e00dc48 fffcac30 Instructions: (pc=0xf6fb1029) 0xf6fb1009: 8b 45 10 89 42 0c 8d 42 18 89 42 10 5d c3 90 90 0xf6fb1019: 8d b4 26 00 00 00 00 55 89 e5 8b 55 08 8b 45 0c 0xf6fb1029: 89 02 c7 42 04 00 00 00 00 c7 42 08 00 00 00 00 0xf6fb1039: 8b 45 10 89 42 0c 8d 42 18 89 42 10 5d c3 90 90 ;; f6fb1019 8d b4 26 00 00 00 00 lea 0x0(%esi,1),%esi ;; f6fb1020 55 push %ebp ;; f6fb1021 89 e5 mov %esp,%ebp ;; f6fb1023 8b 55 08 mov 0x8(%ebp),%edx ;; f6fb1026 8b 45 0c mov 0xc(%ebp),%eax ;; --------------- ;; f6fb1029 89 02 mov %eax,(%edx) ;; f6fb102b c7 42 04 00 00 00 00 movl $0x0,0x4(%edx) ;; f6fb1032 c7 42 08 00 00 00 00 movl $0x0,0x8(%edx) ;; f6fb1039 8b 45 10 mov 0x10(%ebp),%eax ;; f6fb103c 89 42 0c mov %eax,0xc(%edx) ;; f6fb103f 8d 42 18 lea 0x18(%edx),%eax ;; f6fb1042 89 42 10 mov %eax,0x10(%edx) ;; f6fb1045 5d pop %ebp ;; f6fb1046 c3 ret ;; f6fb1047 90 nop ;; f6fb1048 90 nop ;; Register to memory mapping: EAX=0x00003300 is an unknown value EBX=0x9d5f8b30 is an unknown value ECX=0x000032aa is an unknown value EDX=0xffff6000 is an unknown value ESP=0xf77313d8 is pointing into the stack for thread: 0xf6807c00 EBP=0xf77313d8 is pointing into the stack for thread: 0xf6807c00 ESI=0xffff6000 is an unknown value EDI=0x00003300 is an unknown value Stack: [0xf76e2000,0xf7733000], sp=0xf77313d8, free space=316k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x683029] Metachunk::Metachunk(unsigned int, VirtualSpaceNode*)+0x9;; Metachunk::Metachunk(unsigned int, VirtualSpaceNode*)+0x9 V [libjvm.so+0x687e18] VirtualSpaceList::get_new_chunk(unsigned int, unsigned int, unsigned int)+0x38;; VirtualSpaceList::get_new_chunk(unsigned int, unsigned int, unsigned int)+0x38 V [libjvm.so+0x6895f4] SpaceManager::grow_and_allocate(unsigned int)+0x264;; SpaceManager::grow_and_allocate(unsigned int)+0x264 V [libjvm.so+0x6897d1] SpaceManager::allocate_work(unsigned int)+0x131;; SpaceManager::allocate_work(unsigned int)+0x131 V [libjvm.so+0x689841] SpaceManager::allocate(unsigned int)+0x61;; SpaceManager::allocate(unsigned int)+0x61 V [libjvm.so+0x68998f] Metaspace::allocate(ClassLoaderData*, unsigned int, bool, MetaspaceObj::Type, Thread*)+0xaf;; Metaspace::allocate(ClassLoaderData*, unsigned int, bool, MetaspaceObj::Type, Thread*)+0xaf V [libjvm.so+0x690017] Method::clone_with_new_data(methodHandle, unsigned char*, int, unsigned char*, int, Thread*)+0x427;; Method::clone_with_new_data(methodHandle, unsigned char*, int, unsigned char*, int, Thread*)+0x427 V [libjvm.so+0x7678b7] Relocator::insert_space_at(int, int, unsigned char*, Thread*)+0x1c7;; Relocator::insert_space_at(int, int, unsigned char*, Thread*)+0x1c7 V [libjvm.so+0x5bd44d] VM_RedefineClasses::rewrite_cp_refs_in_method(methodHandle, methodHandle*, Thread*)+0x51d;; VM_RedefineClasses::rewrite_cp_refs_in_method(methodHandle, methodHandle*, Thread*)+0x51d V [libjvm.so+0x5be0c3] VM_RedefineClasses::rewrite_cp_refs_in_methods(instanceKlassHandle, Thread*)+0xe3;; VM_RedefineClasses::rewrite_cp_refs_in_methods(instanceKlassHandle, Thread*)+0xe3 V [libjvm.so+0x5be656] VM_RedefineClasses::rewrite_cp_refs(instanceKlassHandle, Thread*)+0x26;; VM_RedefineClasses::rewrite_cp_refs(instanceKlassHandle, Thread*)+0x26 V [libjvm.so+0x5c2f66] VM_RedefineClasses::merge_cp_and_rewrite(instanceKlassHandle, instanceKlassHandle, Thread*)+0x536;; VM_RedefineClasses::merge_cp_and_rewrite(instanceKlassHandle, instanceKlassHandle, Thread*)+0x536 V [libjvm.so+0x5c3ea3] VM_RedefineClasses::load_new_class_versions(Thread*)+0x443;; VM_RedefineClasses::load_new_class_versions(Thread*)+0x443 V [libjvm.so+0x5c46e7] VM_RedefineClasses::doit_prologue()+0xa7;; VM_RedefineClasses::doit_prologue()+0xa7 V [libjvm.so+0x8683ff] VMThread::execute(VM_Operation*)+0x16f;; VMThread::execute(VM_Operation*)+0x16f V [libjvm.so+0x58ee02] JvmtiEnv::RedefineClasses(int, _jvmtiClassDefinition const*)+0x32;; JvmtiEnv::RedefineClasses(int, _jvmtiClassDefinition const*)+0x32 V [libjvm.so+0x553c68] jvmti_RedefineClasses+0x128;; jvmti_RedefineClasses+0x128 C [libredefineClasses.so+0xb280] Java_runtime_ParallelClassLoading_shared_ClassLoadingController_redefineClasses+0x1a0;; Java_runtime_ParallelClassLoading_shared_ClassLoadingController_redefineClasses+0x1a0 j runtime.ParallelClassLoading.shared.ClassLoadingController.redefineClasses(Ljava/lang/ClassLoader;Lruntime/ParallelClassLoading/shared/ClassLoadingController$ByteCodeForRedefinitionHandler;)I+0 j runtime.ParallelClassLoading.shared.ClassLoadingController.startLoadingIterator()Z+806 j runtime.ParallelClassLoading.shared.ClassLoadingController.runIt([Ljava/lang/String;Ljava/io/PrintStream;)I+9 j runtime.ParallelClassLoading.shared.PrivateMLetController.run([Ljava/lang/String;Ljava/io/PrintStream;)I+9 j runtime.ParallelClassLoading.shared.PrivateMLetController.main([Ljava/lang/String;)V+4 v ~StubRoutines::call_stub V [libjvm.so+0x4adbff] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x101f;; .L362+0x849 V [libjvm.so+0x6eb479] os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x19;; os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x19 V [libjvm.so+0x4ac1cf] JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x2f;; JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x2f V [libjvm.so+0x4f2c8f] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x38f;; jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x38f V [libjvm.so+0x4fdbbf] jni_CallStaticVoidMethod+0xdf;; jni_CallStaticVoidMethod+0xdf C [libjli.so+0x71bf] JavaMain+0x91f;; JavaMain+0x91f C [libpthread.so.0+0x6a49] abort@@GLIBC_2.0+0x6a49 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j runtime.ParallelClassLoading.shared.ClassLoadingController.redefineClasses(Ljava/lang/ClassLoader;Lruntime/ParallelClassLoading/shared/ClassLoadingController$ByteCodeForRedefinitionHandler;)I+0 j runtime.ParallelClassLoading.shared.ClassLoadingController.startLoadingIterator()Z+806 j runtime.ParallelClassLoading.shared.ClassLoadingController.runIt([Ljava/lang/String;Ljava/io/PrintStream;)I+9 j runtime.ParallelClassLoading.shared.PrivateMLetController.run([Ljava/lang/String;Ljava/io/PrintStream;)I+9 j runtime.ParallelClassLoading.shared.PrivateMLetController.main([Ljava/lang/String;)V+4 v ~StubRoutines::call_stub --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0xa088a800 JavaThread "Service Thread" daemon [_thread_blocked, id=3305, stack(0xa062e000,0xa067f000)] 0xa0888800 JavaThread "C1 CompilerThread14" daemon [_thread_blocked, id=3299, stack(0x9ea7f000,0x9eb00000)] 0xa0887400 JavaThread "C1 CompilerThread13" daemon [_thread_blocked, id=3298, stack(0x9ec7f000,0x9ed00000)] 0xa0885c00 JavaThread "C1 CompilerThread12" daemon [_thread_blocked, id=3296, stack(0x9ee7f000,0x9ef00000)] 0xa0884000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=3295, stack(0x9f07f000,0x9f100000)] 0xa0882800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=3294, stack(0x9f27f000,0x9f300000)] 0xa0881400 JavaThread "C2 CompilerThread9" daemon [_thread_blocked, id=3293, stack(0x9f47f000,0x9f500000)] 0xa087fc00 JavaThread "C2 CompilerThread8" daemon [_thread_blocked, id=3292, stack(0x9f67f000,0x9f700000)] 0xa087e400 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=3291, stack(0x9f87f000,0x9f900000)] 0xa087cc00 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=3290, stack(0x9fa7f000,0x9fb00000)] 0xa087b400 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=3289, stack(0x9fc7f000,0x9fd00000)] 0xa087a000 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=3285, stack(0x9fe7f000,0x9ff00000)] 0xa0878800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=3282, stack(0xa007f000,0xa0100000)] 0xa0877000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=3281, stack(0xa027f000,0xa0300000)] 0xa0875800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3280, stack(0xa047f000,0xa0500000)] 0xa0874000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3279, stack(0xa067f000,0xa0700000)] 0xa0bc7800 JavaThread "VM JFR Buffer Thread" daemon [_thread_blocked, id=3273, stack(0xa0c0a000,0xa0c5b000)] 0xa1ce3c00 JavaThread "JFR request timer" daemon [_thread_blocked, id=3267, stack(0xa100d000,0xa105e000)] 0xa1cde800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3266, stack(0xa105e000,0xa10af000)] 0xa1cdd000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=3265, stack(0xa10af000,0xa1100000)] 0xa1cb6800 JavaThread "Finalizer" daemon [_thread_blocked, id=3263, stack(0xe6b2e000,0xe6b7f000)] 0xa1cb5000 JavaThread "Reference Handler" daemon [_thread_blocked, id=3262, stack(0xe6d2e000,0xe6d7f000)] =>0xf6807c00 JavaThread "Main Thread" [_thread_in_vm, id=3220, stack(0xf76e2000,0xf7733000)] Other Threads: 0xa1cb1c00 VMThread [stack: 0xa1412000,0xa1493000] [id=3261] 0xa088c400 WatcherThread [stack: 0x9e87f000,0x9e900000] [id=3306]
25-11-2013