Bug ID: JDK-8319055 JCMD should not buffer the whole output of commands

JDK-8319055 : JCMD should not buffer the whole output of commands

Type: Enhancement
Component: hotspot
Sub-Component: svc

Priority: P4
Status: Resolved
Resolution: Fixed

Submitted: 2023-10-28
Updated: 2025-04-11
Resolved: 2025-03-13

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 25
25 b15Fixed

Related Reports

Causes :	JDK-8352163 - [AIX] SIGILL in AttachOperation::ReplyWriter::write_fully after 8319055
Causes :	JDK-8352180 - AttachListenerThread causes many tests to timeout on Windows
Causes :	JDK-8352117 - Test StreamingOutputTest.java intermittent fails java.lang.NoClassDefFoundError: jdk/test/lib/Utils
Relates :	JDK-8336723 - Add infrastructure to allow jcmd commands in Java to stream output back to the tool
Relates :	JDK-8353479 - jcmd with streaming output breaks intendation
Relates :	JDK-8319307 - DCmds should not assert on truncation and should report truncation
Relates :	JDK-8352392 - AIX: implement attach API v2 and streaming output
Relates :	JDK-8353485 - Jcmd should allow to specify streaming_output mode
Relates :	JDK-8354460 - Streaming output for attach API should be turned on by default

Description

JCMD, using the AttachOperation protocol, sends the whole operation result in one go (first the result -- a 32-bit integer at the beginning of the InputStream, then any additional text output). That precludes using JCMD for any operation that may generate much data.

https://github.com/openjdk/jdk/blob/0aaec97527ddf2b229a9dd6beaa7ff55c635dee5/src/jdk.attach/share/classes/sun/tools/attach/HotSpotVirtualMachine.java#L381

Today this causes:
- delays, sometimes long, if a command produces a lot of data. User gets no feedback, jcmd seemingly hangs
- In release builds, we truncate the output without any indication
- In debug builds, we assert (I am trying to change this, but discussions are bogged down on this)

It would be good if JCMD would stream output data as they become available. Currently DCmd in the HotSpot code uses the C++ bufferedStream class to buffer the entire output of a jcmd, and write the output into the socket in one go.

One consideration is backward compatibility. If this change requires changes in both jcmd and hotspot, we still should be able to use new/old jcmds with new/old JVMs (meaning the ability to couple any jcmd from any jdk with any jvm). Possibly by introducing some sort of legacy mode.

It may be that this is not fixable at all, or only at a cost that is too high. If that is the case, at least we then know.

Comments

This causes crashes on AIX: JDK-8352163
17-03-2025
Changeset: cd1be917 Branch: master Author: Alex Menkov <amenkov@openjdk.org> Date: 2025-03-13 20:05:49 +0000 URL: https://git.openjdk.org/jdk/commit/cd1be9175714186b8881a4d08628fdfcc9382bbc
13-03-2025
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk/pull/23405 Date: 2025-01-31 23:23:36 +0000
31-01-2025
Attach client tools (jcmd, jmap, jstack, etc) enqueue attach operation request, read response code and if it's zero (JNI_OK) read the rest of the reply data and print it to the stdout. So client code it "streaming ready" (and the code is the same from early releases, so there are no compatibility issues here). POC which updates server part only so attach operation handlers parse attach operation requests, sends response code and then writes output without buffering shown that the approach works fine with attach tools (including tools from old releases), but causes a number of test failures (mostly timeouts) . Most failed tests uses 2 scenarios: 1. Do self-attach. In the case test executes client code as well, so server side send some data, but client side cannot read it (many attach handlers execute code at safepoint); When socket/pipe buffer is full, the tests hung; 2. Run attach tool to attach to the main test process. The tests capture the tool output, but output redirection also has a buffer and when the buffer is full (as the test cannot read the output) the tool hangs trying to print to stdout. This test scenarios can be considered as a misuse of attach API (and self-attach is disabled by default), but many tests (jdk, jck, third-party) use the technique, so it doesn't look good to disable it completely. Only client side knows if self-attach is performed, so we need a way for clients to disable streaming output. Suggested solution: - update attach operation handlers to set operation result after command and argument parsing (it would be better to set result after argument validation, but this requires changes in DCmd framework); - implement support for both streaming and buffered output by AttachListener; - update attach protocol to support operation request options; for now the only supported option is "streaming"; - update "getversion" command to report supported command options (so tools can detect it); - update client part to disable streaming output for self-attach; - introduce system properties to set default value for streaming options: (1) for client: value used to attach; (2) for VM: the value is used if client does not specified the option (for example if tool from old release attaches to the VM); On the 1st step both properties should "false" by default (to be switched to "true" after extensive testing of the feature and fixing tests which fails with streaming output) To enable streaming output client tools need to be run with (1) option set to "true"
30-01-2025
We could implement streaming through chunking. Current protocol is: [result]\n[output data] New protocol can be: [result]\space[n bytes of data]\n[n bytes of output data]} These messages can be repeated, because the reader knows how many bytes they have to read before the next metadata message occurs. The \space and \n are required to delimit the different parts. I use \space and \n because this is a text protocol. We add a special-case for End-Of-Message: 0\space-1\n That is, a message with [result] = 0 and [n bytes of data] = -1 means that we're done. I'm working a bit on this, frankly not happy with the amount of copy-and-paste that we have between our different implementations.
11-04-2024
the connection is initiated from https://github.com/openjdk/jdk/blob/0aaec97527ddf2b229a9dd6beaa7ff55c635dee5/src/jdk.attach/linux/classes/sun/tools/attach/VirtualMachineImpl.java#L165-L170 // connected - write request // <ver> <cmd> <args...> try { writeString(s, PROTOCOL_VERSION); // == "1" writeString(s, cmd); which gets checked in https://github.com/openjdk/jdk/blob/0aaec97527ddf2b229a9dd6beaa7ff55c635dee5/src/hotspot/os/linux/attachListener_linux.cpp#L291-L299 if (str_count == 1) { if ((strlen(buf) != strlen(ver_str)) \|\| (atoi(buf) != ATTACH_PROTOCOL_VER)) { // == 1 char msg[32]; os::snprintf_checked(msg, sizeof(msg), "%d\n", ATTACH_ERROR_BADVERSION); write_fully(s, msg, strlen(msg)); return nullptr; } So in JDK 23, we can upgrade PROTOCOL_VERSION to 2. jcmd will first try to connect with PROTOCOL_VERSION==2. if the target VM is 22 or older, it will return ATTACH_ERROR_BADVERSION --> in this case, jcmd closes the connection. It then retries with with PROTOCOL_VERSION==1 Otherwise, the target VM will return multi-part data using PROTOCOL_VERSION==2.
20-02-2024