JDK-8234595 : JfrBuffer::reinitialize failed "assert(!lease()) failed: invariant"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jfr
  • Affected Version: 14,16
  • Priority: P3
  • Status: Resolved
  • Resolution: Cannot Reproduce
  • OS: windows
  • CPU: x86_64
  • Submitted: 2019-11-21
  • Updated: 2020-10-13
  • Resolved: 2020-10-13
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 16
16Resolved
Related Reports
Relates :  
JDK-8248485 - Poor scalability in JfrCheckpointManager when using many threads after JDK-8242008
Description
The following test failed an assertion in the JDK14 CI:

jdk/jfr/jcmd/TestJcmdDumpGeneratedFilename.java

Here's a snippet from the log file:

#section:main
----------messages:(5/284)----------
command: main jdk.jfr.jcmd.TestJcmdDumpGeneratedFilename
reason: User specified action: run main/othervm jdk.jfr.jcmd.TestJcmdDumpGeneratedFilename 
Mode: othervm [/othervm specified]
Additional options from @modules: --add-modules jdk.jfr,jdk.jcmd
elapsed time (seconds): 10.47
----------configuration:(3/47)----------
Boot Layer
  add modules: jdk.jfr jdk.jcmd

----------System.out:(18/1061)*----------
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=t:/workspace/open/src/hotspot/share/jfr/recorder/storage/jfrBuffer.cpp:58
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (t:/workspace/open/src/hotspot/share/jfr/recorder/storage/jfrBuffer.cpp:58), pid=6836, tid=32824
#  assert(!lease()) failed: invariant
#
# JRE version: Java(TM) SE Runtime Environment (14.0+24) (fastdebug build 14-ea+24-1107)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 14-ea+24-1107, mixed mode, sharing, tiered, compressed oops, g1 gc, windows-amd64)
# Core dump will be written. Default location: T:\\testoutput\\test-support\\jtreg_open_test_jdk_jdk_svc\\scratch\\6\\hs_err_pid6836.mdmp
#
# An error report file with more information is saved as:
# T:\\testoutput\\test-support\\jtreg_open_test_jdk_jdk_svc\\scratch\\6\\hs_err_pid6836.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
----------System.err:(0/0)----------
----------rerun:(43/5379)*----------

Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x000000eb415e4800):  JavaThread "JFR Recorder Thread" daemon [_thread_in_vm, id=32824, stack(0x000000eb41ba0000,0x000000eb41ca0000)]

Stack: [0x000000eb41ba0000,0x000000eb41ca0000]
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0xa96171]  os::platform_print_native_stack+0xf1  (os_windows_x86.cpp:369)
V  [jvm.dll+0xc93d0b]  VMError::report+0xf0b  (vmerror.cpp:725)
V  [jvm.dll+0xc955be]  VMError::report_and_die+0x8ae  (vmerror.cpp:1533)
V  [jvm.dll+0xc95cb4]  VMError::report_and_die+0x64  (vmerror.cpp:1317)
V  [jvm.dll+0x4b6e62]  report_vm_error+0x102  (debug.cpp:264)
V  [jvm.dll+0x6a71d5]  JfrBuffer::reinitialize+0x45  (jfrbuffer.cpp:58)
V  [jvm.dll+0x6ab17d]  ReleaseOp<JfrMemorySpace<JfrBuffer,JfrMspaceSequentialRetrieval,JfrCheckpointManager> >::process+0x1ad  (jfrmemoryspace.inline.hpp:429)
V  [jvm.dll+0x6ac426]  JfrCheckpointManager::write_type_set+0x406  (jfrcheckpointmanager.cpp:442)
V  [jvm.dll+0x6dbae4]  JfrRecorderService::_write+0x244  (jfrrecorderservice.cpp:490)
V  [jvm.dll+0x6dd0bf]  JfrRecorderService::rotate+0x39f  (jfrrecorderservice.cpp:456)
V  [jvm.dll+0x6df43e]  recorderthread_entry+0x12e  (jfrrecorderthreadloop.cpp:75)
V  [jvm.dll+0xc3951e]  JavaThread::run+0x27e  (thread.cpp:1954)
V  [jvm.dll+0xc2fa22]  Thread::call_run+0x192  (thread.cpp:403)
V  [jvm.dll+0xa9493e]  thread_native_entry+0x10e  (os_windows.cpp:465)
C  [ucrtbase.DLL+0x203ba]
C  [KERNEL32.DLL+0x13f2]
C  [ntdll.dll+0x154f4]
Comments
Very intermittently seen. The assertion has been removed in JDK-8248485 as part of other changes.
13-10-2020
This is an (almost) impossible situation to occur. During the just completed safepoint, the global epoch was shifted. This provides the JFR Recorder thread with exclusive access to the _free_list_mspace of the JfrCheckpointManager upon safepoint release. Before writing out the data in the _free_list_mspace, the JFR Recorder thread will use storage on the _free_list_mspace (acquired by leasing the buffer) as part of JfrTypeSet::serialize(). Post JfrTypeSet::serialize(), as the JfrCheckpointWriter(s) go out of scope, they release leased buffers back onto the _free_list_mspace, with sequence buffer->clear_lease() (resetting the lease flag) and buffer->release() (releasing the identity claim; this will publize the buffer as available). // this is the problematic buffer from the .mdmp file. dt JfrBuffer 000000eb`413588f0 jvm!JfrBuffer +0x000 _next : 0x000000eb`412d8880 JfrBuffer +0x008 _prev : (null) +0x010 _identity : (null) +0x018 _pos : 0x000000eb`413589b8 "hread???" +0x020 _top : 0x000000eb`4135897f "" +0x028 _flags : 0 +0x02a _header_size : 0x30 +0x02c _size : 0x10000 The assert failed as part of JfrBuffer::reinitialize(), and this would indicate that _flags == 0x4 to signify a LEASE. Since the value is 0 in the core / .mdmp, this is reminiscent of a concurrent race situation. But since there are no concurrent accesses to this buffer at this location, a race would be an impossibility.
05-12-2019
ILW = HLH = P2
26-11-2019