JDK-8226919 : attach in linux hangs due to permission denied accessing /proc/pid/root
  • Type: Bug
  • Component: core-svc
  • Sub-Component: tools
  • Affected Version: 10,11.0.3,13,14
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2019-06-27
  • Updated: 2024-03-01
  • Resolved: 2024-02-09
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 23
23 b10Fixed
Related Reports
Duplicate :  
Relates :  
Relates :  
Relates :  
Relates :  
Description
ADDITIONAL SYSTEM INFORMATION :
Debian Linux 64-bit 9.5
OpenJDK 11.0.4 (11.0.3 wasn't viable because of JDK-8222754)

A DESCRIPTION OF THE PROBLEM :
Beginning with JDK-8179498 attach was changed to locate the socket using a path relative to /proc/pid/root to support attach inside of a container. However, dereferencing this symlink is governed by ptrace access mode PTRACE_MODE_READ_FSCREDS which may not succeed when running as the user running the JRE. This breaks running jcmd and jmap as the same user the JVM is running as. 

In our case /tmp/.java_pid##### is accessible as the user running the JVM so earlier versions of the command-line tools are able to see the JVM even on the same system:
user@host:~# /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/jcmd -l
9378 sun.tools.jcmd.JCmd -l
5837
user@host:~# /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/jcmd -l
9408 jdk.jcmd/sun.tools.jcmd.JCmd -l

In both cases process 5837 is still running and is visible to the output of jps -l:
user@host:~$ /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/jps -l
10056 sun.tools.jps.Jps
5837
user@host:~$ /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/jps -l
5837
10079 jdk.jcmd/sun.tools.jps.Jps

and the Java 8 jcmd is able to send commands to the 5837 process:
user@host:~$ /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/jcmd 5837 VM.version
5837:
OpenJDK 64-Bit Server VM version 11.0.4+4-post-Debian-1
JDK 11.0.4

because /tmp/.java_pid##### is resolvable by user.

Attempting to invoke jmap or jcmd including the PID directly instead of searching for it first results in the following:
Exception in thread "main" com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /proc/5837/root/tmp/.java_pid5837: target process 5837 doesn't respond within 10500ms or HotSpot VM not loaded
        at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:100)
        at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
        at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
        at jdk.jcmd/sun.tools.jmap.JMap.executeCommandForPid(JMap.java:128)
        at jdk.jcmd/sun.tools.jmap.JMap.dump(JMap.java:196)
        at jdk.jcmd/sun.tools.jmap.JMap.main(JMap.java:114)

Additional details: this JRE is created using JNI from an application that forks and changes uid to run the child Java process as a different user with a small set of retained capabilities (using Apache commons-daemon) so the current workaround we're looking into is if changing that pattern is effective but until then this represents a serious regression for us.

REGRESSION : Last worked in version 8u212

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1) Launch Apache commons daemon 1.1.0 as root with argument `-user <user>` with a Java class that implements the Daemon interface and Java 10+ in JAVA_HOME
2) Run jps -v as <user> to locate PID
3) Run jcmd -l as <user> to see PID is missing
4) Run jmap -dump:live,format=b,file=/home/user/heap.hprof 5837  as <user> and see the AttachNotSupportedException

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
jcmd -l shows the expected  PID; jmap generates a heap dump
ACTUAL -
jcmd -l is missing the PID
jmap exits after logging AttachNotSupportedException

---------- BEGIN SOURCE ----------
https://github.com/apache/commons-daemon/blob/master/src/samples/SimpleDaemon.java
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
None yet known

FREQUENCY : always



Comments
Changeset: ac4607ed Author: Sebastian Lövdahl <sebastian.lovdahl@hibox.tv> Committer: Kevin Walls <kevinw@openjdk.org> Date: 2024-02-09 18:32:09 +0000 URL: https://git.openjdk.org/jdk/commit/ac4607ed81eb75f43e7d1062e38506972738d086
09-02-2024

A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/17628 Date: 2024-01-30 10:47:22 +0000
30-01-2024

Some recent mailing list discussion: https://mail.openjdk.org/pipermail/serviceability-dev/2023-May/048293.html
04-05-2023

The attach API uses /proc/pid/root in order to support containers. However, the problem in this bug report seems to be running without containers. I.e., our feature for supporting containers has caused a regression when containers are NOT used. When containers are not used, ns_pid should be the same as pid in the following function. However, they might have been different in by the user that reported this bug. // On Linux a simple handshake is used to start the attach mechanism // if not already started. The client creates a .attach_pid<pid> file in the // target VM's working directory (or temp directory), and the SIGQUIT handler // checks for the file. private File createAttachFile(int pid, int ns_pid) throws IOException { String fn = ".attach_pid" + ns_pid; String path = "/proc/" + pid + "/cwd/" + fn; File f = new File(path); try { // Do not canonicalize the file path, or we will fail to attach to a VM in a container. f.createNewFile(); } catch (IOException x) { String root; if (pid != ns_pid) { // A process may not exist in the same mount namespace as the caller. // Instead, attach relative to the target root filesystem as exposed by // procfs regardless of namespaces. root = "/proc/" + pid + "/root/" + tmpdir; } else { root = tmpdir; } f = new File(root, fn); f.createNewFile(); } return f; }
13-10-2021