JDK-8290043 : serviceability/attach/ConcAttachTest.java failed "guarantee(!CheckJNICalls) failed: Attached JNI thread exited without being detached"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 20,21
  • Priority: P3
  • Status: Open
  • Resolution: Unresolved
  • OS: linux
  • CPU: x86_64,aarch64
  • Submitted: 2022-07-09
  • Updated: 2024-11-22
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 24
24Unresolved
Related Reports
Relates :  
JDK-8225690 - Multiple AttachListener threads can be created
Sub Tasks
JDK-8292061 :  
ProblemList serviceability/attach/ConcAttachTest.java on linux-all - Resolved
Description
The following test failed in the JDK20 CI:

serviceability/attach/ConcAttachTest.java

Here's a snippet from the log file:

----------stdout:(21/1864)----------
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/virtualMemoryTracker.cpp:363
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S8422/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6e89d8b9-911d-425b-9920-73a9bd0caece/runs/93ec339e-22f4-47ee-8e3a-ddf8e7c3dd20/workspace/open/src/hotspot/share/services/virtualMemoryTracker.cpp:363), pid=1381980, tid=1391565
#  guarantee(!CheckJNICalls) failed: Attached JNI thread exited without being detached
#
# JRE version: Java(TM) SE Runtime Environment (20.0+6) (fastdebug build 20-ea+6-268)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 20-ea+6-268, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x1b60ff7]  VirtualMemoryTracker::add_reserved_region(unsigned char*, unsigned long, NativeCallStack const&, MEMFLAGS)+0x1037
#
# Core dump will be written. Default location: /opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S19281/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6eacd903-d30d-438a-ba0a-9ab1aca97499/runs/7ce86dcf-cf49-464a-ac9e-92e9c6ca9f68/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1/scratch/3/core
#
# An error report file with more information is saved as:
# /opt/mach5/mesos/work_dir/slaves/0c72054a-24ab-4dbb-944f-97f9341a1b96-S19281/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/6eacd903-d30d-438a-ba0a-9ab1aca97499/runs/7ce86dcf-cf49-464a-ac9e-92e9c6ca9f68/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1/scratch/3/hs_err_pid1381980.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
result: Error. Agent communication error: java.io.EOFException; check console log for any additional details


Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x00007f73c003e1b0):  JavaThread "process reaper" daemon [_thread_new, id=1391565, stack(0x00007f73fc8e9000,0x00007f73fc90f000)]

Stack: [0x00007f73fc8e9000,0x00007f73fc90f000],  sp=0x00007f73fc90dce0,  free space=147k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1b60ff7]  VirtualMemoryTracker::add_reserved_region(unsigned char*, unsigned long, NativeCallStack const&, MEMFLAGS)+0x1037
V  [libjvm.so+0x1a9f10f]  ThreadStackTracker::new_thread_stack(void*, unsigned long, NativeCallStack const&)+0x4f
V  [libjvm.so+0x1a876f8]  Thread::register_thread_stack_with_NMT()+0x88
V  [libjvm.so+0x1a88ded]  Thread::call_run()+0x7d
V  [libjvm.so+0x174bf14]  thread_native_entry(Thread*)+0x104
Comments
We have this in virtualMemoryTracker.cpp: // Overlapped reservation. // It can happen when the regions are thread stacks, as JNI // thread does not detach from VM before exits, and leads to // leak JavaThread object if (reserved_rgn->flag() == mtThreadStack) { guarantee(!CheckJNICalls, "Attached JNI thread exited without being detached"); // Overwrite with new region ... Is that guarantee correct? There's not necessarily a bug in NMT as much as there might an outdated guarantee.
22-11-2024
Reopening the ticket in order to avoid a failure related to ProblemListing.
26-07-2024
[~dholmes], thank you for explaining, I wasn't aware of that. I'm re-opening the ticket.
26-07-2024
[~jsjolen] we can't just close this as it is associated with a ProblemListed test, and that will now cause the PL verification logic to flag this as a malformed PL entry. I would suggest just keeping this open and then close as duplicate if the VMATree rewrite does fix it.
26-07-2024
Will most probably be fixed when VirtualMemoryTracker has been ported to use VMATree
25-07-2024
Continuing with my comment 2 above regarding SIGQUIT and all the extra thread dumps I see in the LingeredApp output, I think I know what is going on. It looks like SIGQUIT is only needed when the socket file (.java_pid<pid>) doesn't already exist. Once a SIGQUIT is sent, it should get created and subsequent attaches won't need to send a SIGQUIT again. However, when there are multiple attaches racing to be the first attach, they will all send a SIGQUIT and wait for the file to be created. The first SIGQUIT to arrive will start the process of starting up the listener thread and creating the socket file, and all the other SIGQUITs probably are ignored by the attach mechanism and are passed through to the exception handler to do the thread dump. Generally speaking that means that whenever you do an attach, you might end up triggering a thread dump if someone else is trying to do the same at the same time (and this is the first attach to the process).
09-11-2023
There's also a minor bug in the test. It does the "vm = VirtualMachine.attach()" in a try block and then "vm.detach()" in the finally block. The problem is if the attach() failed then vm will be null, resulting in an NPE. So the test fails with the NPE rather than the exception thrown during the attach() call. There's no need for the detach() (or anything else) to be in the finally block.
08-11-2023
One interesting thing I just noticed about linux runs of this test is that stderr for the LingeredApp contains a bunch of thread dumps. Initially they can be confused for the dump you get from the Thread.print jcmd that the test is also doing, but since this is LingeredApp output and not the main test output, it can't be from the jcmd. I believe these thread dumps being generated due to SIGQUITs being received. The Attach API sends a SIGQUIT to initiate the attach handshake. I believe it retries if there is no response within a short period of time. So I'm guessing some of these initial SIGQUIT handshakes are not being handled by the attach listener and instead are passed on to the SIGQUIT handler, which does the thread dump. Eventually the Attach API retries with another SIGQUIT that is successful. This might be normal behavior when doing a bunch of simultaneous attaches. Maybe the first SIGQUIT triggers the starting of the listener thread, and subsequent SIGQUITS are missed until the listener thread is done starting up. However, I'm only seeing it Linux, not OSX.
08-11-2023
I would be more suspicious of a bug relating to virtual memory tracking than having a thread actually exit without detaching.
09-08-2022
This intermittent test failure occurs in tier4-rt-check-jni test tasks on linux-aarch64 and linux-x64. The -Xcheck:jni option is catching the fact that the "process reaper" daemon thread is in VirtualMemoryTracker::add_reserved_region() for its stack and runs into an overlap with an existing reserved region. That should only happen when a JNI attached thread has exited without detaching. Since the -Xcheck:jni option is specified, the guarantee() catches this situation and we crash to report the JNI issue/problem. It's not clear what JNI attached thread has exited from the log files.
08-08-2022
ILW = HLL = P4
12-07-2022