JDK-8299426 : Heap dump does not contain virtual Thread stack references
  • Type: Bug
  • Component: hotspot
  • Sub-Component: svc
  • Affected Version: 19,21
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2022-12-26
  • Updated: 2023-12-11
  • Resolved: 2023-12-10
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 23 Other
23 masterFixed naResolved
Related Reports
Cloners :  
Relates :  
Relates :  
Relates :  
Relates :  
Sub Tasks
JDK-8316691 :  
Description
ADDITIONAL SYSTEM INFORMATION :
Ubunu 22.10 x86_64
openjdk version "21-ea" 2023-09-19
OpenJDK Runtime Environment (build 21-ea+3-124)
OpenJDK 64-Bit Server VM (build 21-ea+3-124, mixed mode, sharing)

A DESCRIPTION OF THE PROBLEM :
HPROF files do not contain stack references of unmounted VirtualThreads. This would be necessary for all analyses based on reachability like biggest objects and dominator trees.
The HPROF file format generally supports variable-length HPROF_GC_INSTANCE_DUMP records so the references could be appended to the StackChunk dump.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
With the attached source, create a HPROF file and open it in a HPROF analyser that can show incoming references.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
There should be a reference to the VThreadReferenced instance.
ACTUAL -
There are no references to the VThreadReferenced instance.

---------- BEGIN SOURCE ----------
-- HprofVThreadStackRef.java ---------------------------------------------------------
import com.sun.management.HotSpotDiagnosticMXBean;

import java.io.File;
import java.lang.management.ManagementFactory;
import java.util.concurrent.CountDownLatch;

public class HprofVThreadStackRef {
    public static void main(String[] args) throws Exception {
        CountDownLatch dumpedLatch = new CountDownLatch(1);
        Thread vthread = Thread.ofVirtual().start(() -> {
            Object referenced = new VThreadReferenced();
            System.out.println(referenced.getClass());
            await(dumpedLatch);
            System.out.println(referenced.getClass());
        });
        Thread.sleep(2000); // wait for reference and unmount
        HotSpotDiagnosticMXBean hotSpotDiagnosticMXBean = ManagementFactory.getPlatformMXBean(HotSpotDiagnosticMXBean.class);
        File hprofFile = new File("vthread_test.hprof");
        hprofFile.delete();
        hotSpotDiagnosticMXBean.dumpHeap(hprofFile.getPath(), true);
        dumpedLatch.countDown();
        vthread.join();
    }

    private static void await(CountDownLatch dumpedLatch) {
        try {
            dumpedLatch.await();
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }

    public static class VThreadReferenced {
    }
}
--------------------------------------------------------------------------------------

---------- END SOURCE ----------

FREQUENCY : always



Comments
[~amenkov] Can you manually Resolve this as "Fixed" in build "master" now?
08-12-2023

Fix was pushed while main issue was targeted to 22. Reset the main issue to fixed in 23 and copied the Robo Duke entry here.
08-12-2023

Changeset: 354ea4c2 Author: Alex Menkov <amenkov@openjdk.org> Date: 2023-12-07 23:18:23 +0000 URL: https://git.openjdk.org/jdk/commit/354ea4c28f1449479f71e89831c64047c50e1a61
08-12-2023

ILW=MMM=P3
05-12-2023

A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/16665 Date: 2023-11-14 21:54:06 +0000
14-11-2023

The issue is not about VirtualThread objects, but about absence of thread stack references for unmounted virtual threads. I.e. in heap dump stack objects have less incoming references than they actually have (and if an object is referenced only from unmounted vthread stack like in the testcase provided, in the dump it looks like unreferenced object). Suggested by submitter way to add references as artifical fields of StackChunk object does not look good to me (it would also require to add fictive HPROF_GC_CLASS_DUMP subrecord to describe additional fields). Additional thoughts: - as heap dump is usually used for heap analysis, difference between virtual and platform threads look nonessential; - description of HPROF_GC_ROOT_THREAD_OBJ (there is no official hprof spec, only description in source files) says "thread object" and doesn't mention GC roots; - HPROF_HEAP_DUMP subrecords don't have info about size, so addition of new subrecord type would break existing parsers; we can add new record types (records outside of HPROF_HEAP_DUMP records) as they have info about size and hprof readers skip (at lease should skip) unknown records; Current plan is to handle virtual threads the same way as platform threads: - for each virtual thread write HPROF_GC_ROOT_THREAD_OBJ subrecord, report stack references by HPROF_GC_ROOT_JAVA_FRAME; HPROF_GC_ROOT_THREAD_OBJ requires stack trace sequence number (HPROF_TRACE records). We can write HPROF_TRACE/HPROF_FRAME records for unmounted vthreads during heap scan using "segmented heap dump" feature or we can use dummy STACK_TRACE_ID record (empty stack trace record). If later we decide that we need a way to differentiate platform/virtual threads, new record type can be introduced.
14-09-2023

In addition to what David said, a virtual thread that is sleeping will be kept alive by the timer mechanism. A virtual thread that is parked on I/O will be kept alive by the I/O mechanism. So there will be references in the heap to virtual threads for the heap dump. I'm not 100% sure on if HPROF_GC_ROOT_THREAD_OBJ make sense for virtual threads. A virtual thread is not a GC root so might be misleading if tools reading a heap dump treated it as such. There isn't anything in the THREAD_OBJ sub-record to say if a thread is virtual or not. That said, a possible starting point is to change VM_HeapDumper::do_thread to only walk the carrier thread stack. Next, would be to special case VirtualThread objects and emit a HPROF_GC_ROOT_THREAD_OBJ sub-record - this would need to walk the virtual thread stack for both the mounted and unmounted cases so there are JAVA_FRAME sub-records stack locals, and JNI_LOCAL sub-records for the mounted case.
13-02-2023

Ok. It looks from the VirtualThread you can get to the Continuation, and from there to the StackChunk. GC (and heap dumpers) just need to recognize they are scanning a StackChunk and look at the bits not visible from java to find all the references. I don't think we necessarily need to have VirtualThreads be handled as HPROF_GC_ROOT_THREAD_OBJ. They can probably just be dumped like any ordinary object. However, the HPROF_GC_ROOT_JAVA_FRAME record will need the "thread serial number" for the thread, and it won't have one unless there is an HPROF_GC_ROOT_THREAD_OBJ record. Not sure how that might impact hprof readers.
10-01-2023

If a virtual thread is executing then it is found via the carrier thread. If it is blocked/parked on a synchronization object then it is found through that synchronization object.
10-01-2023

I guess we should consider how GC does a liveness scan. If virtual threads are not roots, there must still be a way for GC to find live VirtualThread objects and StackChunk objects. How does it currently do that?
09-01-2023

[~cjplummer] I'm not sure yet if we can get away with using the existing sub-records or whether we will have to rev the format. I think the main issue is that a virtual thread is not a GC root so I don't think there will be a HPROF_GC_ROOT_THREAD_OBJ for virtual threads. We might be able to get away with using HPROF_GC_ROOT_JAVA_FRAME and HPROF_GC_ROOT_JNI_LOCAL for the references from the virtual thread stack. The stack trace for the mounted virtual thread should be separate from the carrier's stack trace. Similar work is required for the JVMTI FollowReference function as that wasn't included in the first phase of JVMTI work. This has been noticed by one of the profiler maintainers and reported as JDK-8299414.
07-01-2023

[~alanb] I think you are suggesting that a StackChunk should be represented as series of HPROF_GC_ROOT_JAVA_FRAME records, and since HPROF_GC_ROOT_JAVA_FRAME records reference HPROF_GC_ROOT_THREAD_OBJ records, we would also need an HPROF_GC_ROOT_THREAD_OBJ for each virtual thread. I'm not so sure if anything is needed related to HPROF_GC_ROOT_JNI_LOCAL. Is it possible to have a native frames for unmounted virtual threads? BTW, I'm guessing currently for mounted virtual threads the hprof file includes the combined carrier thread + virtual thread stack trace. In other words, it doesn't know about virtual threads. So work is needed in this area also.
06-01-2023

SA has the same issue: JDK-8261848: LOOM: SA needs to add support for scanning StackChunks
05-01-2023

Targeted to 21 for now to keep it under radar. At least, it has to be evaluated.
05-01-2023

The heap dump wasn't prioritised in JDK 19 or 20 but it does need attention. One issue is that virtual Threads are not GC roots so need to decide if HPROF_GC_ROOT_THREAD_OBJ is okay to emit for a virtual Thread object or whether a new record type is needed. Same thing for the HPROF_GC_ROOT_JAVA_FRAME and HPROF_GC_ROOT_JNI_LOCAL sub-records.
31-12-2022