JDK-8179498 : attach in linux should be relative to /proc/pid/root and namespace aware
  • Type: Enhancement
  • Component: core-svc
  • Sub-Component: tools
  • Affected Version: 8,9,10
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • Submitted: 2017-05-02
  • Updated: 2019-07-18
  • Resolved: 2017-09-26
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 10
10 b31Fixed
Related Reports
Relates :  
Relates :  
Relates :  
Sub Tasks
JDK-8187592 :  
JDK-8196878 :  
Description
From TJ Fontaine:

Diagnostic commands (i.e. jcmd, jstack, etc) fail to attach to a target JVM
that is inside a container (e.g. Docker).

A Linux container often isolates a process in a PID and Mount namespace that is separate from the "root container" (analogous to the hypervisor/dom0 in hardware virtualization environments, or the global zone on Solaris). A target JVM that is isolated in either a PID namespace, or a Mount namespace will fail the attach sequence.

When the target JVM is in its own PID namespace the pid of the process is distinct from what the real pid of the process as it relates to the root
container. For example, in the root container you can observe a JVM with a pid of 17734, however if that JVM is running inside a Docker container the pid inside its PID namespace is likely 1. So when the target JVM receives the SIGQUIT it looks in /proc/self/cwd/ for .attach_pid1 however the external attaching JVM has created the file /proc/17734/cwd/.attach_pid17734. Given this discrepancy the target JVM will output to stderr thread status, since /proc/self/cwd/.attach_pid1 doesn't exist and won't continue with the attach sequence.

The solution is to parse /proc/pid/status for the field NSpid (available since Linux 4.1) which contains a list of pids, where the last entry is the "innermost" PID namespace value. (Namespaces can be stacked, unlike Solaris Zones which have a virtualization depth of 1)

The rest of the Linux attach sequence assumes a shared mount namespace by waiting for /tmp/.java_pid17734 to appear. But if the attaching process is in a separate namespace because the target JVM is in a mount namepsace (or in a chroot as well) the unix domain socket for attaching won't appear.

Instead the attach sequence should resolve file names relative to
/proc/17734/root which has a materialized view of the rootfs for the target.

Comments
The patch referenced from the above discussion does work, but there is still one issue not resolved by it. What the patch solves is mapping the JVM's PID as the host sees to the PID as the docker sees it, and then properly creating the /tmp/.java_pid<pid> file files using the PID as the docker sees it. This resolves the attach protocol issues. What it doesn't resolve is the hosts inability to find all JVMs running on the docker. "jcmd -l" relies on this, as does "jcmd <mainclass>" to fine the JVM that is running <mainclass>. Normally this JVM list is obtained by looking in /tmp/hsperfdata_<user>/* to find the list of JVM pid's for each user. However, since the docker maps to a different /tmp, jcmd on the host does not see these hsperfdata files for docker JVMs. I think resolving this would require some additional docker knowledge being added. If you can get the list of PIDs for the docker (or at least for the primordial process), then you can get to the tmp dir using "/proc/<pid>/root/tmp/", and look for hsperfdata files there. However, this is beyond the scope of this CR.
08-09-2017

Discussion thread: http://mail.openjdk.java.net/pipermail/serviceability-dev/2017-May/021249.html
02-05-2017