Bug ID: JDK-8259526 "Reference Handler" idle thread has misleading thread state

Type: Enhancement
Component: core-svc
Sub-Component: java.lang.management
Affected Version: 17

Priority: P4
Status: Open
Resolution: Unresolved

Submitted: 2021-01-10
Updated: 2022-02-01

Other
tbdUnresolved

I noticed a small usability regression apparently introduced by JDK-8156500.
When the ReferenceHandler thread is idle (i.e. almost always) its stack trace misleadingly suggests it is RUNNABLE

"Reference Handler" daemon prio=10 Id=2 RUNNABLE
at java.base@14-ea/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
at java.base@14-ea/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
at java.base@14-ea/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)

it would be nice if the user-visible java thread state and lock info was the same as if it was blocked inside Object.wait, giving stack traces something like

"Reference Handler" daemon prio=10 Id=2 WAITING on VM heap lock

which would make "dump all threads" output less confusing.

I agree there are no easy answers. Java users are frustrated that the thread state isn't telling them what they want, e.g. when blocking in native read(). Java can't tell the user whether the thread is "really" RUNNABLE, nor even that the thread is in some UNKNOWN NATIVE state. If we had a thread state for NATIVE, then for many applications NATIVE would be largely synonymous with "blocked on I/O". We added an optional blocker to park methods, but in order for that to work any blocking must be done via park and not some lower level code in the VM. Can Reference Handler thread some day be a Loom virtual thread?
18-01-2021
The GC question here was "should we change the way the blocking of the reference handler thread is done?". The information is not coming from the VM runtime but the ThreadInfo instance - hence sending this over to core-svc for consideration. But there is no simple fix here as the information is simply not available via the existing APIs, so we have two choices if we want to "fix" this: 1. Add a new state to Thread.State to cover native blocking states and use that within the VM and other library APIs that do native blocking. This would be a fairly major RFE, potentially JEP-worthy. 2. Expand j.l.management.ThreadInfo to include information about the native thread state. This is also not a small RFE as this is an exported JDK API and we'd need to carefully consider how to expose any information in a platform independent way.
18-01-2021
Kicking this back to runtime. This issue has no specific GC content. It is a generic issue/question about how the status should be reported for a thread blocked on a VM monitor. It's possible that it should be considered not an issue and no change should be made. But that's not a GC question, it's a runtime (or perhaps serviceability) question. It might also be considered an enhancement rather than a bug.
15-01-2021
Okay so unfortunately what you are seeing is the information held by java.lang.management.ThreadInfo, and that doesn't include any detailed native thread state, only the java.lang.Thread state. So there is no way the VM can make any change to what is printed here. The only way to change the RUNNABLE state is to change the code so that it isn't actually RUNNABLE - which would basically require undoing the change to move the blocking into the VM. Or file an enhancement to add additional states to Thread.State that could then be used for such cases. Other thread dumps (jstack, ctrl-/) show the native thread information as previously listed.
14-01-2021
IIRC I noticed the State change looking at the output from THREAD_MXBEAN.dumpAllThreads(true, true)) https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/concurrent/tck/JSR166TestCase.java#L1172
13-01-2021
This is no different to any thread blocked in native I/O showing that it is Runnable. We don't have Java Thread.State states that map to all possible blocking conditions that may be encountered, so we can't just set the state to something other than RUNNABLE. That is unfortunate. [~martin] It would be quite painful to have the ReferenceHandler thread use a Java-level blocking mechanism that ends up in park(), because the state it is checking needs to be something that the VM exposes, and the VM then needs to emulate the execution of Java code that would do the notification. I don't think the VM can actually execute Java code to do this because the GC threads can't execute Java code. [~tschatzl] it is possible to print the native osThread/JavaThread state in addition to the java.lang.Thread state, but I don't think we have the necessary information to give a detailed accounting. At present if your print a thread dump via ctrl-/ then you would see: "Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.21ms elapsed=3.50s tid=0x00007f31e8288800 nid=0x2c4e waiting on condition [0x00007f31a86b6000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList(java.base@14/Native Method) at java.lang.ref.Reference.processPendingReferences(java.base@14/Reference.java:241) at java.lang.ref.Reference$ReferenceHandler.run(java.base@14/Reference.java:213) which shows that the thread is waiting on a condition, even though its Java state is RUNNABLE. [~martin] exactly how was this "dump all threads" done? The Java APIs for this tend to produce Java-developer friendly output and avoid OS/native thread details.
13-01-2021
The way that this is architected elsewhere in the java libraries is via a call to one of the park methods, which the VM is aware of. Not sure if Reference Handler's java code can be changed to use park somehow, which would fix the Thread.State.
11-01-2021
The ReferenceHandler thread is blocking in native since JDK-8156500, and the current thread printing code can not provide this information. Kindly asking the runtime team typically working in this area (i.e. JavaThread::print_on()) to investigate whether it is feasible to add support for some "WAITING on XYZ lock" for locks held by VM code. Also changing to enhancement due to scope change.
11-01-2021