JDK-8316239 : JFR: fatal error: refcount has gone to zero
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jfr
  • Affected Version: 22,23
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • Submitted: 2023-09-14
  • Updated: 2025-05-26
  • Resolved: 2025-05-26
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 25
25 masterResolved
Related Reports
Blocks :  
JDK-8352251 - Implement JEP 518: JFR Cooperative Sampling
Duplicate :  
JDK-8352251 - Implement JEP 518: JFR Cooperative Sampling
Relates :  
JDK-8321822 - ZGC: SIGSEGV in "JFR Recorder Thread"
Description
Stress test crashes with 
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (c:\sb\prod\1693840994\workspace\open\src\hotspot\share\oops\symbol.cpp:317), pid=1476, tid=38628
#  fatal error: refcount has gone to zero
#
# JRE version: Java(TM) SE Runtime Environment (22.0+14) (fastdebug build 22-ea+14-1002)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 22-ea+14-1002, mixed mode, sharing, tiered, compressed class ptrs, z gc, windows-amd64)
# Core dump will be written. Default location: C:\sb\prod\1694182672\testoutput\test-support\jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink7D_java\scratch\0\hs_err_pid1476.mdmp
#
# JFR recording file will be written. Location: C:\sb\prod\1694182672\testoutput\test-support\jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink7D_java\scratch\0\hs_err_pid1476.jfr
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Xbootclasspath/a:C:\sb\prod\1694182672\testoutput\test-support\jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink7D_java\scratch\0\wb.jar -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -Djava.security.manager=allow -XX:MaxRAMPercentage=4.16667 -Dtest.boot.jdk=c:\ade\mesos\work_dir\jib-master\install\jdk\20\36\bundles\windows-x64\jdk-20_windows-x64_bin.zip\jdk-20 -Djava.io.tmpdir=c:\sb\prod\1694182672\testoutput\test-support\jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink7D_java\tmp -XX:+CreateCoredumpOnCrash -XX:+UseZGC -XX:MaxRAMPercentage=50 -XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:+CrashOnOutOfMemoryError -Djava.net.preferIPv6Addresses=false -XX:+DisplayVMOutputToStderr -Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags -XX:+DisableExplicitGC -XX:+StartAttachListener --enable-preview --add-exports=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-exports=java.base/jdk.internal.org.objectweb.asm=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED -Djava.io.tmpdir=C:\sb\prod\1694182672\testoutput\test-support\jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink7D_java\scratch\0/java.io.tmpdir -Duser.home=C:\sb\prod\1694182672\testoutput\test-support\jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink7D_java\scratch\0/user.home -agentpath:c:\ade\mesos\work_dir\jib-master\install\jdk-22+14-1002\windows-x64-debug.test\hotspot\jtreg\native\JvmtiStressModule.dll -XX:NativeMemoryTracking=detail -Djdk.test.lib.random.seed=-1953753881415736653 applications.kitchensink.process.stress.Main C:\sb\prod\1694182672\testoutput\test-support\jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink7D_java\scratch\0\kitchensink.final.properties

Host: win2022-x64-154089, AMD EPYC 7J13 64-Core Processor                , 12 cores, 23G,  Windows Server 2022 , 64 bit Build 20348 (10.0.20348.1668)
Time: Thu Sep 14 01:31:15 2023 /GM elapsed time: 468637.740251 seconds (5d 10h 10m 37s)

---------------  T H R E A D  ---------------

Current thread (0x0000028c2f18e530):  JavaThread "JFR Recorder Thread" daemon [_thread_in_vm, id=38628, stack(0x00000090dd700000,0x00000090dd800000) (1024K)]

Stack: [0x00000090dd700000,0x00000090dd800000]
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0xc76981]  os::win32::platform_print_native_stack+0x101  (os_windows_x86.cpp:236)
V  [jvm.dll+0xf1801f]  VMError::report+0x147f  (vmError.cpp:1000)
V  [jvm.dll+0xf1a485]  VMError::report_and_die+0x645  (vmError.cpp:1814)
V  [jvm.dll+0x54b1fe]  report_fatal+0x7e  (debug.cpp:212)
V  [jvm.dll+0xe305f9]  Symbol::increment_refcount+0x79  (symbol.cpp:317)
V  [jvm.dll+0x853883]  HashTableHost<Symbol const *,unsigned __int64,ListEntry,JfrSymbolTable,1009>::put+0x123  (jfrHashtable.hpp:208)
V  [jvm.dll+0x8534b6]  JfrSymbolTable::mark+0xb6  (jfrSymbolTable.cpp:203)
V  [jvm.dll+0x8532f5]  JfrSymbolTable::mark+0x195  (jfrSymbolTable.cpp:298)
V  [jvm.dll+0x867ee7]  write_klass+0xe7  (jfrTypeSet.cpp:220)
V  [jvm.dll+0x86622d]  JfrArtifactCallbackHost<Klass const *,CompositeFunctor<Klass const *,JfrTypeWriterHost<JfrPredicatedTypeWriterImplHost<Klass const *,SerializePredicate<Klass const *>,&write__klass>,180>,KlassArtifactRegistrator> >::do_artifact+0x2d  (jfrTypeSetUtils.hpp:73)
V  [jvm.dll+0x85e3f3]  EpochDispatchOp<JfrEpochQueue<JfrEpochQueueKlassPolicy>::ElementDispatch<KlassFunctor> >::dispatch+0x133  (jfrStorageUtils.inline.hpp:171)
V  [jvm.dll+0x85dc66]  JfrLinkedList<JfrBuffer,JfrCHeapObj>::iterate<EpochDispatchOp<JfrEpochQueue<JfrEpochQueueKlassPolicy>::ElementDispatch<KlassFunctor> > >+0x76  (jfrLinkedList.inline.hpp:83)
V  [jvm.dll+0x85dd99]  JfrEpochQueue<JfrEpochQueueKlassPolicy>::iterate<KlassFunctor>+0xd9  (jfrEpochQueue.inline.hpp:113)
V  [jvm.dll+0x85eb51]  JfrTraceIdKlassQueue::iterate+0x41  (jfrTraceIdKlassQueue.cpp:269)
V  [jvm.dll+0x8668c3]  do_klasses+0x33  (jfrTypeSet.cpp:360)
V  [jvm.dll+0x868131]  write_klasses+0x141  (jfrTypeSet.cpp:398)
V  [jvm.dll+0x86760e]  JfrTypeSet::serialize+0x5e  (jfrTypeSet.cpp:1071)
V  [jvm.dll+0x801102]  flush_type_set+0x92  (jfrCheckpointManager.cpp:610)
V  [jvm.dll+0x8011ed]  JfrCheckpointManager::flush_type_set+0x5d  (jfrCheckpointManager.cpp:621)
V  [jvm.dll+0x844c5b]  JfrRecorderService::flush+0x24b  (jfrRecorderService.cpp:650)
V  [jvm.dll+0x844fc7]  JfrRecorderService::invoke_flush+0x147  (jfrRecorderService.cpp:664)
V  [jvm.dll+0x844dfe]  JfrRecorderService::flushpoint+0x4e  (jfrRecorderService.cpp:675)
V  [jvm.dll+0x846b9c]  recorderthread_entry+0x1bc  (jfrRecorderThreadLoop.cpp:85)
V  [jvm.dll+0x7f8f19]  JavaThread::thread_main_inner+0x279  (javaThread.cpp:721)
V  [jvm.dll+0xe8220d]  Thread::call_run+0x1cd  (thread.cpp:225)
V  [jvm.dll+0xc747b2]  os::win32::thread_native_entry+0xa2  (os_windows.cpp:545)
C  [ucrtbase.dll+0x26b4c]  (no source info available)
C  [KERNEL32.DLL+0x14dd0]  (no source info available)
C  [ntdll.dll+0x7e3db]  (no source info available)
Comments
Here's the crashing thread's stack from the jdk-23+5-296-tier7 sighting: applications/runthese/RunThese30M.java --------------- T H R E A D --------------- Current thread (0x00007f7fa8e71260): JavaThread "JFR Recorder Thread" daemon [_thread_in_vm, id=261455, stack(0x00007f805dd1f000,0x00007f805de1f000) (1024K)] Stack: [0x00007f805dd1f000,0x00007f805de1f000], sp=0x00007f805de1d7c0, free space=1017k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xe650f7] Symbol::increment_refcount()+0xa7 (symbol.cpp:316) V [libjvm.so+0x9761b9] JfrSymbolTable::mark(unsigned long, Symbol const*, bool)+0x109 (jfrSymbolTable.cpp:141) V [libjvm.so+0x98ceac] write_klass(JfrCheckpointWriter*, Klass const*, bool, int&)+0x11c (jfrTypeSet.cpp:104) V [libjvm.so+0x99333d] JfrArtifactCallbackHost<Klass const*, CompositeFunctor<Klass const*, CompositeFunctor<Klass const*, JfrTypeWriterHost<JfrPredicatedTypeWriterImplHost<Klass const*, LeakPredicate<Klass const*>, &(write__klass__leakp(JfrCheckpointWriter*, void const*))>, 182u>, JfrTypeWriterHost<JfrPredicatedTypeWriterImplHost<Klass const*, SerializePredicate<Klass const*>, &(write__klass(JfrCheckpointWriter*, void const*))>, 182u> >, KlassArtifactRegistrator> >::do_artifact(void const*)+0x5d (jfrTypeSet.cpp:388) V [libjvm.so+0x984186] EpochDispatchOp<JfrEpochQueue<JfrEpochQueueKlassPolicy>::ElementDispatch<KlassFunctor> >::dispatch(bool, unsigned char const*, unsigned long)+0x96 (jfrTraceIdKlassQueue.hpp:42) V [libjvm.so+0x984256] void JfrLinkedList<JfrBuffer, JfrCHeapObj>::iterate<CompositeOperation<EpochDispatchOp<JfrEpochQueue<JfrEpochQueueKlassPolicy>::ElementDispatch<KlassFunctor> >, ReinitializeAllReleaseRetiredOp<JfrMemorySpace<JfrEpochStorageHost<JfrBuffer, JfrMspaceRemoveRetrieval, false>, JfrMspaceRemoveRetrieval, JfrConcurrentQueue<JfrBuffer, JfrCHeapObj>, JfrLinkedList<JfrBuffer, JfrCHeapObj>, true>, JfrLinkedList<JfrBuffer, JfrCHeapObj> >, CompositeOperationAnd> >(CompositeOperation<EpochDispatchOp<JfrEpochQueue<JfrEpochQueueKlassPolicy>::ElementDispatch<KlassFunctor> >, ReinitializeAllReleaseRetiredOp<JfrMemorySpace<JfrEpochStorageHost<JfrBuffer, JfrMspaceRemoveRetrieval, false>, JfrMspaceRemoveRetrieval, JfrConcurrentQueue<JfrBuffer, JfrCHeapObj>, JfrLinkedList<JfrBuffer, JfrCHeapObj>, true>, JfrLinkedList<JfrBuffer, JfrCHeapObj> >, CompositeOperationAnd>&)+0x56 (jfrStorageUtils.inline.hpp:160) V [libjvm.so+0x983da9] JfrTraceIdKlassQueue::iterate(void (*)(Klass*), bool)+0x129 (jfrMemorySpace.inline.hpp:297) V [libjvm.so+0x9845d0] JfrTraceIdLoadBarrier::do_klasses(void (*)(Klass*), bool)+0x20 (jfrTraceIdLoadBarrier.cpp:108) V [libjvm.so+0x990aae] JfrTypeSet::serialize(JfrCheckpointWriter*, JfrCheckpointWriter*, bool, bool)+0x27e (jfrTypeSet.cpp:553) V [libjvm.so+0x91724d] JfrCheckpointManager::write_type_set()+0xfd (jfrCheckpointManager.cpp:614) V [libjvm.so+0x967bb8] JfrRecorderService::post_safepoint_write()+0x78 (jfrRecorderService.cpp:607) V [libjvm.so+0x968c26] JfrRecorderService::rotate(int)+0xa6 (jfrRecorderService.cpp:565) V [libjvm.so+0x969161] recorderthread_entry(JavaThread*, JavaThread*)+0x111 (jfrRecorderThreadLoop.cpp:81) V [libjvm.so+0x90b4f8] JavaThread::thread_main_inner() [clone .part.0]+0xb8 (javaThread.cpp:721) V [libjvm.so+0xebd37f] Thread::call_run()+0x9f (thread.cpp:220) V [libjvm.so+0xce1535] thread_native_entry(Thread*)+0xd5 (os_linux.cpp:789)
12-01-2024
I think this bug could be solved by the proposed patch in JDK-8321098
04-12-2023
The JFR thread sampler has recorded the following stacktrace: dt JfrStackTrace 0000028c`543d6c40 jvm!JfrStackTrace +0x000 _next : (null) +0x008 _frames : 0x0000028c`53646d50 JfrStackFrame +0x010 _id : 0x13527e4 +0x018 _hash : 0x5dd2e363`93ebb6cf +0x020 _nr_of_frames : 3 +0x024 _max_frames : 0x40 +0x028 _frames_ownership : 1 +0x029 _reached_root : 1 +0x02a _lineno : 1 +0x02b _written : 1 It contains 3 frames: 0:055> dt 0x0000028c`53646d50 JfrStackFrame jvm!JfrStackFrame +0x000 _klass : 0x0000028c`0f239730 InstanceKlass +0x008 _methodid : 0x5a10024 +0x010 _line : 0n382 +0x014 _bci : 0n19 +0x018 _type : 0x2 '' <<-- JIT inlined dt 0x0000028c`53646d70 JfrStackFrame jvm!JfrStackFrame +0x000 _klass : 0x0000028c`0f239730 InstanceKlass +0x008 _methodid : 0x5a10005 +0x010 _line : 0n916 +0x014 _bci : 0n2 +0x018 _type : 0x1 '' <<-- JIT compiled 0:055> dt 0x0000028c`53646d90 JfrStackFrame jvm!JfrStackFrame +0x000 _klass : 0x0000028c`104ee800 InstanceKlass +0x008 _methodid : 0x00000021`c4fb0000 +0x010 _line : 0n5 +0x014 _bci : 0n0 +0x018 _type : 0x1 '' <<-- JIT Compiled (0) java/util/TreeMap.getKey(Object key):line=382, bci=0n19, frame_type=JIT Inlined (1) java/util/TreeMap.remove(Object key):line=916, bci=2, frame_type=JIT Compiled (2) applications/kitchensink/process/stress/modules/Dummy1912149.<init>():line=5, bci=0, frame_type=JIT Compiled Frame nr 2 references a Method* of a class that has been unloaded. I believe this is an invalid stack trace not reified in the code. This hypothesis indicates a problem in the stack walker code. It's very unusual to be able to resolve the top methods correctly but eventually run into an invalid method higher up the stack.
01-11-2023
I am running out of ideas from digging about in the .mdmp file. Moving to gc for their input if there could be an issue with concurrent class unloading.
01-11-2023
dt InstanceKlass 0000028c`104ee800 jvm!InstanceKlass =00007fff`a2fe64d8 _shared_metaspace_base : 0x0000028c`0f000000 Void =00007fff`a2fe64e0 _shared_metaspace_top : 0x0000028c`0fd10000 Void +0x000 __VFN_table : 0x00007fff`a2bd4868 +0x008 _valid : 0n0 +0x010 _layout_helper : 0n16 +0x014 _kind : 0 ( InstanceKlassKind ) +0x018 _modifier_flags : 0n1 +0x01c _super_check_offset : 0x40 +0x020 _name : 0x0000028e`79e261e8 Symbol // the name Symbol +0x028 _secondary_super_cache : (null) +0x030 _secondary_supers : 0x0000028c`0f694560 Array<Klass *> +0x038 _primary_supers : [8] 0x0000028c`0f000f80 Klass +0x078 _java_mirror : OopHandle +0x080 _super : 0x0000028c`0f000f80 Klass +0x088 _subklass : (null) +0x090 _next_sibling : 0x0000028c`10400000 Klass +0x098 _next_link : (null) +0x0a0 _class_loader_data : 0x0000028c`3f496260 ClassLoaderData +0x0a8 _vtable_len : 0n6 +0x0ac _access_flags : AccessFlags +0x0b0 _trace_id : 0x00000021`c4fb1005 +0x0b8 _shared_class_path_index : 0n-1 +0x0ba _shared_class_flags : 0 +0x0c0 _annotations : (null) +0x0c8 _package_entry : 0x0000028c`4f7dee90 PackageEntry +0x0d0 _array_klasses : (null) +0x0d8 _constants : 0x0000028c`2cffb028 ConstantPool +0x0e0 _inner_classes : 0x0000028c`0f6949a0 Array<unsigned short> +0x0e8 _nest_members : 0x0000028c`0f6949a0 Array<unsigned short> +0x0f0 _nest_host : (null) +0x0f8 _permitted_subclasses : 0x0000028c`0f6949a0 Array<unsigned short> +0x100 _record_components : (null) +0x108 _source_debug_extension : (null) +0x110 _nonstatic_field_size : 0n0 +0x114 _static_field_size : 0n1 +0x118 _nonstatic_oop_map_size : 0n0 +0x11c _itable_len : 0n2 +0x120 _nest_host_index : 0 +0x122 _this_class_index : 8 +0x124 _static_oop_field_count : 1 +0x126 _idnum_allocated_count : 2 +0x128 _init_state : 5 ( fully_initialized ) +0x129 _reference_type : 0 '' +0x12a _misc_flags : InstanceKlassFlags +0x130 _init_monitor : 0x0000028c`5a893020 Monitor +0x138 _init_thread : (null) +0x140 _oop_map_cache : (null) +0x148 _jni_ids : (null) +0x150 _methods_jmethod_ids : (null) +0x158 _dep_context : (null) +0x160 _dep_context_last_cleaned : 0x1f4d3 +0x168 _osr_nmethods_head : (null) +0x170 _breakpoints : (null) +0x178 _previous_versions : (null) +0x180 _cached_class_file : (null) +0x188 _jvmti_cached_class_field_map : (null) +0x190 _verify_count : 0n0 +0x194 _shared_class_load_count : 0n0 +0x198 _methods : 0x0000028c`2cffb188 Array<Method *> +0x1a0 _default_methods : (null) +0x1a8 _local_interfaces : 0x0000028c`0f694a18 Array<InstanceKlass *> +0x1b0 _transitive_interfaces : 0x0000028c`0f694a18 Array<InstanceKlass *> +0x1b8 _method_ordering : 0x0000028c`2cffb358 Array<int> +0x1c0 _default_vtable_indices : (null) +0x1c8 _fieldinfo_stream : 0x0000028c`2cffb340 Array<unsigned char> +0x1d0 _fields_status : 0x0000028c`2cffb350 Array<FieldStatus> =00007fff`a30394b0 _disable_method_binary_search : 0 =00007fff`a2f75a68 _finalization_enabled : 1 =00007fff`a30394b1 _should_clean_previous_versions : 0 // _name Symbol ((jvm!Symbol *)0x28e79e261e8) : 0x28e79e261e8 [Type: Symbol *] [=0x7fffa2fe64d8] _shared_metaspace_base : 0x28c0f000000 [Type: void *] [=0x7fffa2fe64e0] _shared_metaspace_top : 0x28c0fd10000 [Type: void *] [+0x000] _hash_and_refcount : 0x856d0000 [Type: unsigned int] // <<---- the _name Symbol for the instanceKlass has a refCount of 0. [+0x004] _length : 0x3c [Type: unsigned short] [+0x006] _body [Type: unsigned char [2]] [=0x7fffa305ea20] _vm_symbols [Type: Symbol * [0]] [=0x7fffa3060fc8] _total_count : 0xf81b16e [Type: unsigned __int64] // _class_loader_data ClassLoaderData dx -r1 ((jvm!ClassLoaderData *)0x28c3f496260) ((jvm!ClassLoaderData *)0x28c3f496260) : 0x28c3f496260 [Type: ClassLoaderData *] [=0x7fffa302b580] _the_null_class_loader_data : 0x28c6cd10d80 [Type: ClassLoaderData *] [+0x000] _holder [Type: WeakHandle] [+0x008] _class_loader [Type: OopHandle] [+0x010] _metaspace : 0x0 [Type: ClassLoaderMetaspace *] [+0x018] _metaspace_lock : 0x28c5a8920a0 [Type: Mutex *] [+0x020] _unloading : true [Type: bool] // Unloading == true [+0x021] _has_class_mirror_holder : false [Type: bool] [+0x022] _modified_oops : true [Type: bool] [+0x024] _keep_alive : 0 [Type: int] // _keep_alive == 0 [+0x028] _claim : 0 [Type: int] [+0x030] _handles [Type: ClassLoaderData::ChunkedHandleList] [+0x038] _dependency_count : 0 [Type: int] [+0x040] _klasses : 0x28c104ee800 [Type: Klass *] [+0x048] _packages : 0x0 [Type: PackageEntryTable *] [+0x050] _modules : 0x0 [Type: ModuleEntryTable *] [+0x058] _unnamed_module : 0x0 [Type: ModuleEntry *] [+0x060] _dictionary : 0x0 [Type: Dictionary *] [+0x068] _jmethod_ids : 0x28c37fe2670 [Type: JNIMethodBlock *] [+0x070] _deallocate_list : 0x0 [Type: GrowableArray<Metadata *> *] [+0x078] _next : 0x28c3f4960c0 [Type: ClassLoaderData *] [+0x080] _unloading_next : 0x28c3f4969b0 [Type: ClassLoaderData *] [+0x088] _class_loader_klass : 0x28c10295150 [Type: Klass *] [+0x090] _name : 0x0 [Type: Symbol *] [+0x098] _name_and_id : 0x28c5e76f258 [Type: Symbol *] [+0x0a0] _trace_id : 0xf0a060801 [Type: unsigned __int64]
30-10-2023