JDK-8325621 : Improve jspawnhelper version checks
  • Type: Enhancement
  • Component: core-libs
  • Sub-Component: java.lang
  • Affected Version: 11,17,21,22,23
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2024-02-12
  • Updated: 2025-08-13
  • Resolved: 2024-03-14
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 17 JDK 21 JDK 22 JDK 23
17.0.13Fixed 21.0.4Fixed 22.0.2Fixed 23 b15Fixed
Related Reports
Causes :  
Relates :  
Relates :  
Relates :  
Relates :  
Description
jspawnhelper is invoked by JDK ProcessBuilder/Runtime.exec to invoke a process with posix_spawn. There is the internal protocol between JDK invocation code and jspawnhelper, which changes sometimes, see e.g.  JDK-8310265.

We expect that only JDK invokes jspawnhelper. But there are two corner cases.

First, someone could call jspawnhelper directly. There is a warning message printed in some cases, for example when jspawnhelper is invoked without arguments: "This command is not for general use and should only be run as the result of a call to ProcessBuilder.start() or Runtime.exec() in a java application". Quick googling shows that projects still ignore this warning. See for example Zimbra (https://wiki.zimbra.com/wiki/Jspawnhelper): "One example of legitimate use [sic! -Aleksey] of jspawnhelper is by zmprov, the command line utility to provision accounts on Zimbra."

Second, there is a minuscule chance of system misconfiguration when jspawnhelper from another JDK would be executed instead of one shipping with invoking JDK. Current code figures out the jspawnhelper path from `java.home` property, which looks safe-ish, but it would be even better to make sure. There is also the report from Ubuntu: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280 -- that unattended upgrade replaced the JDK under the running JVM, which effectively replaced the old jspawnhelper with an incompatible one. This is arguably the upgrade process problem, as JDK is not guaranteed to work when its contents are overwritten, but we want to catch that error consistently.

I think we can make jspawnhelper checks a bit better by handshaking on explicit JDK/VM version. Something like `VERSION_STRING` already defined by the build system can be compiled into the `jspawnhelper` and checked against.
Comments
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk17u-dev/pull/2592 Date: 2024-06-15 00:13:47 +0000
13-08-2025

Note: I filed JDK-8352489 to (at least temporarily) relax the version checks to informative.
20-03-2025

@Alexey Shipilev: I agree - we should address the issue in unattended-upgrades[1]/needrestart[2] rather than try to make openjdk gracefully handle overwrites. There is a bug filed against unattended-upgrades[3] and some activity on it might help to prioritise it. I have done some testing with needrestart - it handles Java systemd services, but fails to notify if the Java process is running in background in the terminal session and does not do anything if the runtime is removed. [1] https://launchpad.net/ubuntu/+source/needrestart [2] https://launchpad.net/ubuntu/+source/unattended-upgrades [3] https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280
14-08-2024

I agree that in-place overwrite of JDK binary while Java applications are running is problematic and should be avoided. The ideal approach is ask all package managers and JDK installers to avoid or fix this problem before installing the JDK containing this change. The question is how to safely communicate and roll out this change, without suddenly breaking running Java applications when the package managers upgrade the JDK in the old way that they have been doing for perhaps decades. Should we publish a release note highlighting this compatibility risk? Are there channels or precedence to notify maintainers of supported OS distributions about this potential breaking change?
14-08-2024

Of course, I don't see that email in my inbox, so I cannot reply there :( Yes, jspawnhelper checks would detect the case when JDK was overwritten while JDK was running. This is the expected behavior for this change: it fails fast, instead of giving the cryptically misbehaving JVM. Note that we started doing this _because_ there were cryptic bugs after jspawnhelper bugfixes, and the upgrade overwriting the JDK contents with incompatible jspawnhelper. The upgrades 22.0.1 -> 22.0.2 would still yield not very helpful message about the incorrect number of arguments, but going forward, the overwrite would clearly trigger: ``` Incorrect Java version: 21.0.4-internal-adhoc.shipilev.openjdk-jdk21u-dev jspawnhelper version 21.0.5-internal-adhoc.shipilev.openjdk-jdk21u-dev ``` JDK cannot generally survive overwrite without breaking, even it "appeared" to work well before. The JDK libraries, symbols, etc would also be overwritten, which can result in much subtler bugs when incompatible code is loaded. Upgrade process that either overwrites the JDK contents for running JVMs, or does not suggest to restart JDKs after upgrade is the root issue here, not the jspawnhelper checks themselves. On other words, version checking only highlights the problem with upgrades processes. Package managers should prompt users to restart their Java processes after in-place upgrade, and unattended upgrade should not attempt to upgrade JDK without users knowing about it. Technically, we can revert these version checks and instead just silently accept the overwrite like before. I don't think we should get back to that behavior, though. It does not solve the actual issue, and just sweeps it under the rug. The running Java service is now subtly broken after the overwrite, you just don't know about it, and might be just lucky it does not affect correctness. So I think there is some work needed on distro side. Vladimir Petko (Canonical) was in this thread from the start. Vladimir, what's your thinking here?
14-08-2024

"compatibility impact of JDK-8325621 (Improve jspawnhelper version checks)" https://mail.openjdk.org/pipermail/core-libs-dev/2024-August/127583.html
14-08-2024

[jdk17u-fix-request] Approval Request from Chad Rakoczy Clean backport to update jspawnhelper to check that JDK version and jspawnhelper version are the same. Updates test to include check for version. GHA tests pass including the updated test. Risk is low: touches the production code paths but only passes an additional argument to jspawnhelper.
16-06-2024

[jdk21u-fix-request] Approval Request from Chad Rakoczy Clean backport to update jspawnhelper to check that JDK version and jspawnhelper version are the same. Updates test to include check for version. GHA tests pass including the updated test. This has been backported to JDK22. Risk is low: touches the production code paths but only passes an additional argument to jspawnhelper.
08-05-2024

A pull request was submitted for review. URL: https://git.openjdk.org/jdk22u/pull/181 Date: 2024-05-02 19:14:54 +0000
02-05-2024

[jdk22u-fix-request] Approval Request from Chad Rakoczy Clean backport to update jspawnhelper to check that JDK version and jspawnhelper version are the same. Updates test to include check for version. GHA tests pass including the updated test. Risk is low: touches the production code paths but only passes an additional argument to jspawnhelper
02-05-2024

[21u] Please backport to 22 first. Removing the lable in the meantime.
02-05-2024

A pull request was submitted for review. URL: https://git.openjdk.org/jdk21u-dev/pull/547 Date: 2024-05-01 22:05:45 +0000
01-05-2024

Changeset: a232e8fb Author: Chad Rakoczy <chadrako@amazon.com> Committer: Aleksey Shipilev <shade@openjdk.org> Date: 2024-03-14 13:26:03 +0000 URL: https://git.openjdk.org/jdk/commit/a232e8fb4e6e9e2e9a5285bf01c93b8d1d995f04
14-03-2024

A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/18204 Date: 2024-03-11 17:46:14 +0000
11-03-2024

Yes, if the fixed width is used to transfer the version then it straightforward and does not add any complexity/failure modes. To help the user with the diagnostics, maybe it would make sense to print the version encoded in jspawnhelper when it is called without args?
04-03-2024

Actually, using the major/minor version number is better, using fixed fields and easier reading/writing.
04-03-2024

Including the version string in the handshake is about as straightforward as it can get and does not add any new failure modes or complex code.
04-03-2024

The original requirements for the jspawnhelper state that it has to be as simple as possible in order to be reliable. Returning a version in a keep-alive response is not a complex change, but I was thinking if we can avoid changing jspawnhelper altogether. Renaming the jspawnhelper will require changes in DefaultImageBuilder to rename the file to the expected name and write the property file (if we go with the append a timestamp approach) and this is definitely far more complex than just sending a version string. Though considering the error handling (since the requirements state that the execute-twice approach is for rare cases when normal fork/exec might crash in rare cases): - jspawnhelper does not send anything at all and times out - we can just raise "jspawnhelper timed out" without further details. - jspawnhelper sends some binary junk (e.g. due to the build error) - The message becomes garbled potentially corrupting the logs. - jspawnhelper keeps sending data - we need to limit size of the version string - jspawnhelper crashes mid transfer of the version string - the "expected" part of the message becomes truncated potentially corrupting the log or lacks information, so we can just state that "jspawnhelper failed to start." Regarding the message, in the case of file not found, ProcessBuilder can check {java.home}/lib for the binaries named "jspawnhelper*" and warn the user in the format above "Expected jspawnhelper for Java 21.0.1+13, but jspawnhelper for Java 21.0.1+14 was found". With timestamp it will look something like "Expected jspawnhelper for Java 21.0.1+13 built on 1 Jan 2023, but jspawnhelper for Java 21.0.1+14 built on 2 Jan 2024 was found instead." This has less error handling issues to care about and will provide reasonable information in most cases. Error cases here: - lib directory becomes is not listable - we can raise explicit error message for this - files in lib were deleted and no jspawnhelper* binary found - same - wrong permissions - same - ??? other cases?
01-03-2024

That might work too. I see two wrinkles with that approach: 1) we would need to change build system to accommodate that change; 2) the "file not found" looks like a less understandable failure mode than printing out the "Incompatible JDK version, expected FOO, but got BAR. Check A, B, C." from the jspawnhelper itself.
01-03-2024

I am wondering if this has to be a handshake - maybe it is possible to make jspawnhelper file name build or version-specific? E.g. jspawnhelper<timestamp> or jspawnhelper<version> . This will not protect from malicious replacement (e.g. solarwinds-style attack), but will provide a clear message to the user that the expected file is now missing from the file system.
01-03-2024