JDK-8029178 : Parallel class loading test anonymous-simple gets SIGSEGV in Metaspace::contains
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: hs25
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • Submitted: 2013-11-26
  • Updated: 2014-07-29
  • Resolved: 2014-01-13
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 8 JDK 9
8u20Fixed 9 b02Fixed
Related Reports
Relates :  
Description
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xfffffd625bef230c, pid=14567, tid=92
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b117) (build 1.8.0-ea-fastdebug-b117)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b61-internal-201311222208.amurillo.hs25-b61-gc-sync-fastdebug mixed mode solaris-amd64 compressed oops)

Stack: [0xfffffd7ffa829000,0xfffffd7ffa929000],  sp=0xfffffd7ffa924800,  free space=1006k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x20c230c]  bool Metaspace::contains(const void*)+0x2c;;  __1cJMetaspaceIcontains6Fpkv_b_+0x2c
V  [libjvm.so+0x1057c00]  Klass*Dependencies::DepStream::context_type()+0x208;;  __1cMDependenciesJDepStreamMcontext_type6M_pnFKlass__+0x208
V  [libjvm.so+0x1062085]  Klass*Dependencies::DepStream::check_klass_dependency(KlassDepChange*)+0x20d;;  __1cMDependenciesJDepStreamWcheck_klass_dependency6MpnOKlassDepChange__pnFKlass__+0x20d
V  [libjvm.so+0x21a19f1]  bool nmethod::check_all_dependencies()+0x91;;  __1cHnmethodWcheck_all_dependencies6M_b_+0x91
V  [libjvm.so+0xe23829]  int CodeCache::mark_for_deoptimization(DepChange&)+0x49d;;  __1cJCodeCacheXmark_for_deoptimization6FrnJDepChange__i_+0x49d
V  [libjvm.so+0x27e409c]  void Universe::flush_dependents_on(instanceKlassHandle)+0x80;;  __1cIUniverseTflush_dependents_on6FnTinstanceKlassHandle__v_+0x80
V  [libjvm.so+0x26c179a]  void SystemDictionary::add_to_hierarchy(instanceKlassHandle,Thread*)+0x5a;;  __1cQSystemDictionaryQadd_to_hierarchy6FnTinstanceKlassHandle_pnGThread__v_+0x5a
V  [libjvm.so+0x26bfc3e]  void SystemDictionary::define_instance_class(instanceKlassHandle,Thread*)+0xaee;;  __1cQSystemDictionaryVdefine_instance_class6FnTinstanceKlassHandle_pnGThread__v_+0xaee
V  [libjvm.so+0x26c089f]  instanceKlassHandle SystemDictionary::find_or_define_instance_class(Symbol*,Handle,instanceKlassHandle,Thread*)+0x7e3;;  __1cQSystemDictionarybDfind_or_define_instance_class6FpnGSymbol_nGHandle_nTinstanceKlassHandle_pnGThread__4_+0x7e3
V  [libjvm.so+0x26bc31d]  Klass*SystemDictionary::resolve_from_stream(Symbol*,Handle,Handle,ClassFileStream*,bool,Thread*)+0x73d;;  __1cQSystemDictionaryTresolve_from_stream6FpnGSymbol_nGHandle_3pnPClassFileStream_bpnGThread__pnFKlass__+0x73d
V  [libjvm.so+0x1a3c187]  _jclass*jvm_define_class_common(JNIEnv_*,const char*,_jobject*,const signed char*,int,_jobject*,const char*,unsigned char,Thread*)+0x4c3;;  __1cXjvm_define_class_common6FpnHJNIEnv__pkcpnI_jobject_pkWi53CpnGThread__pnH_jclass__+0x4c3
V  [libjvm.so+0x1a3d677]  JVM_DefineClassWithSource+0x5e3;;  JVM_DefineClassWithSource+0x5e3
C  [libjava.so+0x11cfe]  Java_java_lang_ClassLoader_defineClass1+0x12a;;  Java_java_lang_ClassLoader_defineClass1+0x12a
J 227  java.lang.ClassLoader.defineClass1(Ljava/lang/String;[BIILjava/security/ProtectionDomain;Ljava/lang/String;)Ljava/lang/Class; (0 bytes) @ 0xfffffd7feb4d0ad9 [0xfffffd7feb4d0920+0x1b9]
J 259 C1 java.lang.ClassLoader.defineClass(Ljava/lang/String;[BIILjava/security/ProtectionDomain;)Ljava/lang/Class; (43 bytes) @ 0xfffffd7feb5029e4 [0xfffffd7feb5022c0+0x724]
J 234 C1 java.net.URLClassLoader.defineClass(Ljava/lang/String;Lsun/misc/Resource;)Ljava/lang/Class; (224 bytes) @ 0xfffffd7feb4e140c [0xfffffd7feb4df440+0x1fcc]
J 149 C1 java.net.URLClassLoader$1.run()Ljava/lang/Class; (73 bytes) @ 0xfffffd7feb48e4ac [0xfffffd7feb48df00+0x5ac]
J 115 C1 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) @ 0xfffffd7feb460294 [0xfffffd7feb460200+0x94]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x16ddb72]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x1622;;  __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x1622
V  [libjvm.so+0x16dc50f]  void JavaCalls::call(JavaValue*,methodHandle,JavaCallArguments*,Thread*)+0x3f;;  __1cJJavaCallsEcall6FpnJJavaValue_nMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x3f
V  [libjvm.so+0x1a501e4]  JVM_DoPrivileged+0x167c;;  JVM_DoPrivileged+0x167c
C  [libjava.so+0x10d97]  Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedExceptionAction_2Ljava_security_AccessControlContext_2+0xf;;  Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedExceptionAction_2Ljava_security_AccessControlContext_2+0xf
J 114  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0xfffffd7feb45fee3 [0xfffffd7feb45fd60+0x183]
J 113 C1 java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class; (29 bytes) @ 0xfffffd7feb45f33c [0xfffffd7feb45f000+0x33c]
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+70
J 171 C1 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7 bytes) @ 0xfffffd7feb4a0d14 [0xfffffd7feb4a0c00+0x114]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x16ddb72]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x1622;;  __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x1622
V  [libjvm.so+0x16dc50f]  void JavaCalls::call(JavaValue*,methodHandle,JavaCallArguments*,Thread*)+0x3f;;  __1cJJavaCallsEcall6FpnJJavaValue_nMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x3f
V  [libjvm.so+0x16d98bc]  void JavaCalls::call_virtual(JavaValue*,KlassHandle,Symbol*,Symbol*,JavaCallArguments*,Thread*)+0x77c;;  __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_pnGSymbol_5pnRJavaCallArguments_pnGThread__v_+0x77c
V  [libjvm.so+0x16da27d]  void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Handle,Thread*)+0x149;;  __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_pnGSymbol_63pnGThread__v_+0x149
V  [libjvm.so+0x26be651]  instanceKlassHandle SystemDictionary::load_instance_class(Symbol*,Handle,Thread*)+0x36d;;  __1cQSystemDictionaryTload_instance_class6FpnGSymbol_nGHandle_pnGThread__nTinstanceKlassHandle__+0x36d
V  [libjvm.so+0x26b9980]  Klass*SystemDictionary::resolve_instance_class_or_null(Symbol*,Handle,Handle,Thread*)+0xc8c;;  __1cQSystemDictionarybEresolve_instance_class_or_null6FpnGSymbol_nGHandle_3pnGThread__pnFKlass__+0xc8c
V  [libjvm.so+0x26b4065]  Klass*SystemDictionary::resolve_or_fail(Symbol*,Handle,Handle,bool,Thread*)+0x28d;;  __1cQSystemDictionaryPresolve_or_fail6FpnGSymbol_nGHandle_3bpnGThread__pnFKlass__+0x28d
V  [libjvm.so+0x1ae8489]  _jclass*find_class_from_class_loader(JNIEnv_*,Symbol*,unsigned char,Handle,Handle,unsigned char,Thread*)+0x39;;  __1cbCfind_class_from_class_loader6FpnHJNIEnv__pnGSymbol_CnGHandle_4CpnGThread__pnH_jclass__+0x39
V  [libjvm.so+0x1a3a1b9]  JVM_FindClassFromClassLoader+0x6ad;;  JVM_FindClassFromClassLoader+0x6ad
C  [libjava.so+0x11975]  Java_java_lang_Class_forName0+0xd1;;  Java_java_lang_Class_forName0+0xd1
j  java.lang.Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;+0
j  java.lang.Class.forName(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;+41
j  runtime.ParallelClassLoading.shared.ProvokeType.provoke(Ljava/lang/ClassLoader;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V+50
j  runtime.ParallelClassLoading.shared.ProvokeType.provoke(Ljava/lang/ClassLoader;Ljava/lang/String;)V+8
j  runtime.ParallelClassLoading.shared.ClassLoadingThread.run()V+83
v  ~StubRoutines::call_stub
V  [libjvm.so+0x16ddb72]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x1622;;  __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x1622
V  [libjvm.so+0x16dc50f]  void JavaCalls::call(JavaValue*,methodHandle,JavaCallArguments*,Thread*)+0x3f;;  __1cJJavaCallsEcall6FpnJJavaValue_nMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x3f
V  [libjvm.so+0x16d98bc]  void JavaCalls::call_virtual(JavaValue*,KlassHandle,Symbol*,Symbol*,JavaCallArguments*,Thread*)+0x77c;;  __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_pnGSymbol_5pnRJavaCallArguments_pnGThread__v_+0x77c
V  [libjvm.so+0x16da10d]  void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*)+0xed;;  __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_pnGSymbol_6pnGThread__v_+0xed
V  [libjvm.so+0x1aafabb]  void thread_entry(JavaThread*,Thread*)+0xc7;;  __1cMthread_entry6FpnKJavaThread_pnGThread__v_+0xc7
V  [libjvm.so+0x274a291]  void JavaThread::thread_main_inner()+0x521;;  __1cKJavaThreadRthread_main_inner6M_v_+0x521
V  [libjvm.so+0x27499a7]  void JavaThread::run()+0x84f;;  __1cKJavaThreadDrun6M_v_+0x84f
V  [libjvm.so+0x22b1b12]  java_start+0x1ce;;  java_start+0x1ce
C  [libc.so.1+0x1222ad]  _thrp_setup+0xa5;;  _thrp_setup+0xa5
C  [libc.so.1+0x122550]  _lwp_start+0x0;;  _lwp_start+0x0
Comments
See backport bug.
13-01-2014

The fix version is 8-pool. Changing it to 9 perhaps will fix this.
10-01-2014

I checked this in. Why is it still open? changeset: 5741:ce86c36b8921 user: coleenp date: Tue Jan 07 13:26:56 2014 -0500 summary: 8029178: Parallel class loading test anonymous-simple gets SIGSEGV in Metaspace::contains
10-01-2014

Strange how when it rains it pours. I have a fix. I wonder if we should un-defer this.
02-01-2014

Release team: Approved for deferral.
12-12-2013

Given that this is a rare case, SQE is OK to defer.
11-12-2013

Defer justification: this is an extremely rare race condition in debug code that can only be triggered by CMS (I believe). The proper fix requires cleanup which adds risk.
11-12-2013

'o' seems is not a good InstanceKlass. If hs_err_pid14567 recorded correct information of context contents, rbx contains 'o', which is inside a instanceKlass: 0x0000000100044cc8: 0xfffffd625cfc9008 0x0000000000000000 0x0000000100044cd8: 0x0000004800000020 0x00000000007b3ba8 0x0000000100044ce8: 0x0000000100044ad0 0xfffffd7ffae7e3a8 0x0000000100044cf8: 0x0000000100000fb0 0x000000010001a570 0x0000000100044d08: 0x000000010001e120 0x0000000100044cc8 0x0000000100044d18: 0x0000000000000000 0x0000000000000000 0xfffffd625cfc9008: __1cNInstanceKlassG__vtbl_ : 0xfffffd625cacede8 rbx pointed to second field of InstanceKlass.
10-12-2013

This is really helpful! It looks like Metaspace::contains() is wrong and the comment says why. We added removing virtual space nodes after it was written so traversing the virtual space nodes isn't valid without a lock because one can be removed in the middle. In the hs_err file rbx is: RBX=0x0000000100044cd0 is pointing into metadata This determines that this pointer is in metaspace by traversing the class loader data graph. This cannot be pruned except at safepoint (where it's added to an _unloading list for purging later). So the question is, with CMS is ClassLoaderDataGraph::purge called outside a safepoint?
10-12-2013

Details from core file: thread t@92 thread stack(down): bool Metaspace::contains(const void*)+0x2c; Klass*Dependencies::DepStream::context_type()+0x208; Klass*Dependencies::DepStream::check_klass_dependency(KlassDepChange*)+0x20d; bool nmethod::check_all_dependencies()+0x91; context_type: 0xfffffd625ae87bf2: context_type+0x01fa: testq %r13,%r13 0xfffffd625ae87bf5: context_type+0x01fd: je context_type+0x210 [ 0xfffffd625ae87c08, .+0x13 ] 0xfffffd625ae87bf7: context_type+0x01ff: leaq 0x0000000000000008(%r13),%rdi 0xfffffd625ae87bfb: context_type+0x0203: call is_metaspace_object [ 0xfffffd625a640024, .-0x847bd7 ] 0xfffffd625ae87c00: context_type+0x0208: testl %eax,%eax 0xfffffd625ae87c02: context_type+0x020a: je context_type+0x3b6 [ 0xfffffd625ae87dae, .+0x1ac ] 0xfffffd625ae87c08: context_type+0x0210: testq %r13,%r13 It is calling is_metaspace_object with rdi (this): lost the pointer since core reading failure. But it does not matter here. 0xfffffd625a640024: is_metaspace_object : pushq %rbp 0xfffffd625a640025: is_metaspace_object+0x0001: movq %rsp,%rbp 0xfffffd625a640028: is_metaspace_object+0x0004: leave 0xfffffd625a640029: is_metaspace_object+0x0005: jmp contains [ 0xfffffd625bef22e0, .+0x18b22b7 ] 0xfffffd625a64002e: is_metaspace_object+0x000a: nop This is just save a frame. 0xfffffd625bef22e0: contains : pushq %rbp 0xfffffd625bef22e1: contains+0x0001: movq %rsp,%rbp 0xfffffd625bef22e4: contains+0x0004: pushq %rbx 0xfffffd625bef22e5: contains+0x0005: subq $0x0000000000000008,%rsp 0xfffffd625bef22e9: contains+0x0009: movq %rdi,%rbx 0xfffffd625bef22ec: contains+0x000c: call is_in_shared_space [ 0xfffffd625befbf98, .+0x9cac ] 0xfffffd625bef22f1: contains+0x0011: testl %eax,%eax 0xfffffd625bef22f3: contains+0x0013: jne contains+0xd3 [ 0xfffffd625bef23b3, .+0xc0 ] 0xfffffd625bef22f9: contains+0x0019: movq 0x0000000000fdb208 [ 0xfdb208 ](%rip),%rax 0xfffffd625bef2300: contains+0x0020: movq (%rax),%rax 0xfffffd625bef2303: contains+0x0023: movq 0x0000000000000008(%rax),%rax 0xfffffd625bef2307: contains+0x0027: testq %rax,%rax 0xfffffd625bef230a: contains+0x002a: je contains+0x5a [ 0xfffffd625bef233a, .+0x30 ] 0xfffffd625bef230c: contains+0x002c: movq 0x0000000000000010(%rax),%rcx // <--- Boom here! 0xfffffd625bef2310: contains+0x0030: cmpq %rcx,%rbx 0xfffffd625bef2313: contains+0x0033: jb contains+0x48 [ 0xfffffd625bef2328, .+0x15 ] 0xfffffd625bef2315: contains+0x0035: movq 0x0000000000000018(%rax),%rdx Those frames are optimized results, the calling site for this is: inline Metadata* Dependencies::DepStream::recorded_metadata_at(int i) { Metadata* o = NULL; if (_code != NULL) { o = _code->metadata_at(i); } else { o = _deps->oop_recorder()->metadata_at(i); } assert(o == NULL || o->is_metaspace_object(), // here is the call site for is_metaspace_object()!!! err_msg("Should be metadata " PTR_FORMAT, o)); return o; } Metadata* Dependencies::DepStream::argument(int i) { Metadata* result = recorded_metadata_at(argument_index(i)); // -> call above function Now, look at contains(...): bool Metaspace::contains(const void * ptr) { if (MetaspaceShared::is_in_shared_space(ptr)) { return true; } // This is checked while unlocked. As long as the virtualspaces are added // at the end, the pointer will be in one of them. The virtual spaces // aren't deleted presently. When they are, some sort of locking might // be needed. Note, locking this can cause inversion problems with the // caller in MetaspaceObj::is_metadata() function. return space_list()->contains(ptr) || (using_class_space() && class_space_list()->contains(ptr)); } is_in_shared_space(ptr) returned false. next to check space list: space_list(): static VirtualSpaceList* _space_list; it is a static global variable. It is: 0xfffffd625cff96f8: _space_list : 0x000000000049ea78 Note now, rbx store ptr, we don't know since dbx gives 0 which is not right, but from hs_err recorded value, it is: RBX=0x0000000100044cd0 class VirtualSpaceNode : public CHeapObj<mtClass> { friend class VirtualSpaceList; // Link to next VirtualSpaceNode VirtualSpaceNode* _next; // total in the VirtualSpace MemRegion _reserved; ReservedSpace _rs; VirtualSpace _virtual_space; MetaWord* _top; // count of chunks contained in this VirtualSpace uintx _container_count; ... }; check _space_list: 0x000000000049ea78: 0xfffffd625cff9d50 0x000000000049eae8 vtable _next 0x000000000049ea88: 0x0000000000c0dcb8 0xf1f1f1f1f1f1f100 MemRegion (start) MemRegion(size) 0x000000000049ea98: 0x0000000000640000 0x0000000000635e00 0x000000000049eaa8: 0x0000000000000016 0xabababababababab 0x000000000049eab8: 0xabababababababab 0x00000000000000f1 above is ReservedSpace 0x000000000049eac8: 0x0000000000000000 0xabababababababab 0x000000000049eae8: 0xfffffd625cff9d90 0x0000000000b7d5f8 0x000000000049eaf8: 0xfffffd7ffac00000 0x0000000000100000 0x000000000049eb08: 0xfffffd7ffac00000 0x0000000000800000 0x000000000049eb18: 0x0000000000000000 0x0000000000001000 0x000000000049eb28: 0xfffffd7ffee40000 0xfffffd7ffac00000 looks data already corrupted. If check the pattern 0xf1f1f1..... in RAX, looks like it is set in dtor: VirtualSpaceNode::~VirtualSpaceNode() { _rs.release(); #ifdef ASSERT size_t word_size = sizeof(*this) / BytesPerWord; Copy::fill_to_words((HeapWord*) this, word_size, 0xf1f1f1f1); #endif }
10-12-2013

ILW: HLM -> P3 I: High - crash L: Low, hard to reproduce W: M, not commonly used garbage collector
09-12-2013

Sorry ConcurrentMarkSweep garbage collector.
05-12-2013

What is CMS?
05-12-2013

It's odd that both of these crashes in metaspace are using CMS.
05-12-2013

Host: AMD x86 2593 MHz, 2 cores, 2G, Solaris / Solaris 11, i86pc JDK: Java(TM) SE Runtime Environment 1.8.0 b117 (1.8.0-ea-fastdebug-b117), -d64 -server -Xmixed -XX:MaxRAMFraction=8 -XX:+CreateMinidumpOnCrash -Xconcgc -XX:MaxRAMFraction=8 -XX:+CMSClassUnloadingEnabled -XX:-VerifyBeforeExit
26-11-2013