JDK-8273216 : JCMD does not work across container boundaries with Podman
  • Type: Bug
  • Component: hotspot
  • Sub-Component: svc
  • Affected Version: 11.0.12,17,18
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • Submitted: 2021-09-01
  • Updated: 2024-08-21
  • Resolved: 2021-09-30
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 17 JDK 18
11.0.24-oracleFixed 17.0.11-oracleFixed 18 b18Fixed
Related Reports
Relates :  
JDK-8195809 - [TESTBUG] jps and jcmd -l support for containers is not tested
Description
The test test/hotspot/jtreg/containers/docker/TestJcmd.java that tests how jcmd works across container boundary (invoke from host system to send commands to JVM running inside a container) works well with Docker, but does not work with Podman.

Please note that this test is under development, see: "8195809: [TESTBUG] jps and jcmd -l support for containers is not tested" for details.
Comments
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk17u-dev/pull/2815 Date: 2024-08-19 18:16:12 +0000
19-08-2024
Fix request [17u] I backport this for parity with 17.0.11-oracle. Low risk, as it only involves a test change. Patch is clean. Modified test passes and tier 1 passes with GHA.
19-08-2024
I believe the root cause is to do with a bug associated with the layered fs used by podman, resolved in later releases.
08-03-2024
Changeset: 9180d9a2 Author: Severin Gehwolf <sgehwolf@openjdk.org> Date: 2021-09-30 15:23:44 +0000 URL: https://git.openjdk.java.net/jdk/commit/9180d9a2f990e71ca6ac9c14e55a21f7372929ac
30-09-2021
$ ps ax | grep EventGeneratorLoop 4740 pts/0 Sl 0:00 /jdk/bin/java -cp /test-classes/ EventGeneratorLoop 400 $ jcmd 4740 help 4740: Attach file: /proc/4740/root/tmp/.attach_pid17 java.io.IOException: Permission denied at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method) at java.base/java.io.File.createNewFile(File.java:1043) at jdk.attach/sun.tools.attach.VirtualMachineImpl.createAttachFile(VirtualMachineImpl.java:309) at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:81) at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58) at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207) at jdk.jcmd/sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:113) at jdk.jcmd/sun.tools.jcmd.JCmd.main(JCmd.java:97) This seems to be happening with podman version 3.2.3 and below, but doesn't seem to be happening with podman 3.3.1 and better. The file that fails is /proc/<host-pid>/root/tmp/.attach_pid<container-pid> from this code segment: // On Linux a simple handshake is used to start the attach mechanism // if not already started. The client creates a .attach_pid<pid> file in the // target VM's working directory (or temp directory), and the SIGQUIT handler // checks for the file. private File createAttachFile(int pid, int ns_pid) throws IOException { String fn = ".attach_pid" + ns_pid; String path = "/proc/" + pid + "/cwd/" + fn; File f = new File(path); try { // Do not canonicalize the file path, or we will fail to attach to a VM in a container. f.createNewFile(); } catch (IOException x) { String root; if (pid != ns_pid) { // A process may not exist in the same mount namespace as the caller. // Instead, attach relative to the target root filesystem as exposed by // procfs regardless of namespaces. root = "/proc/" + pid + "/root/" + tmpdir; } else { root = tmpdir; } f = new File(root, fn); // <--- Here f.createNewFile(); } return f; } It works fine for me with podman 3.3.1. So it seems a podman bug which causes this.
28-09-2021
For podman the best approach is to use --userns=keep-id switch when running as root-less. The permission denied error is likely caused by the missing 'z' option to the --volume option.
28-09-2021
should check to see if this bug is manifest against 'runc'/'containerd' which is the actual component responsible for starting the container, delegated to from podman (& docker I believe) Also should check against 'rootless' containers to ensure this works (or determine that when rootless it may well work) should also consider improving the the exception msg to include the file that failed to be created exclusively!
07-09-2021
Failure exception message: jcmd $PID help <PID>: java.io.IOException: Permission denied at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method) at java.base/java.io.File.createNewFile(File.java:1043) at jdk.attach/sun.tools.attach.VirtualMachineImpl.createAttachFile(VirtualMachineImpl.java:307) at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:80) at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58) at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207) at jdk.jcmd/sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:113) at jdk.jcmd/sun.tools.jcmd.JCmd.main(JCmd.java:97)
01-09-2021