JDK-8307977 : jcmd and jstack broken for target processes running with elevated capabilities
  • Type: Bug
  • Component: core-svc
  • Sub-Component: tools
  • Affected Version: 17.0.6
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • OS: linux
  • CPU: x86_64
  • Submitted: 2023-04-17
  • Updated: 2024-02-12
  • Resolved: 2024-02-09
Related Reports
Duplicate :  
Relates :  
Relates :  
Description
ADDITIONAL SYSTEM INFORMATION :
Ubuntu 20.04 and 22.04
OpenJDK 17.0.6+10-Ubuntu-0ubuntu120.04.1 and Temurin 17.0.6+10

A DESCRIPTION OF THE PROBLEM :
Tools like jcmd and jstack do not work if the target process uses elevated capabilities like CAP_NET_BIND_SERVICE (using systemd's AmbientCapabilities or setcap cap_net_bind_service+ep /path/to/java) and when jcmd/jstack is run as the same user as the target process.

This worked fine in Java 8, but since that, the dynamic attach mechanism has been changed from using /tmp/.attach_pidXXXX to /proc/XXXX/root/tmp/.attach_pidXXXX (as seen in current mainline https://github.com/openjdk/jdk/blob/c0b4957fcce530290fe3b1e730b593b6458285aa/src/jdk.attach/linux/classes/sun/tools/attach/VirtualMachineImpl.java#L212-L218).

https://github.com/openjdk/jdk/blob/c0b4957fcce530290fe3b1e730b593b6458285aa/src/jdk.attach/linux/classes/sun/tools/attach/VirtualMachineImpl.java#L212-L218

Using /proc/XXXX/root/tmp/.attach_pidXXXX is a sensible default (see https://bugs.openjdk.org/browse/JDK-8179498 for rationale), I think falling back to /tmp/.attach_pidXXXX outside container environments would make sense.

REGRESSION : Last worked in version 8u361

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
xx@desk:~/reproducer$ sudo apt-get install openjdk-17-jdk-headless
xx@desk:~/reproducer$ cat Reproducer.java 
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.ServerSocket;

public class Reproducer {
  public static void main(String[] args) throws InterruptedException, IOException {
    System.out.println("Hello, World!");
    try (var server = new ServerSocket()) {
      server.bind(new InetSocketAddress("localhost", 81));
      System.out.println("Bound to port 81");
      while (true) {
        Thread.sleep(1_000L);
      }
    }
  }
}

xx@desk:~/reproducer$ cat reproducer.service 
[Service]
Type=simple
ExecStart=/usr/lib/jvm/java-17-openjdk-amd64/bin/java /home/xx/reproducer/Reproducer.java

User=xx
Group=xx
ReadWritePaths=/tmp
AmbientCapabilities=CAP_NET_BIND_SERVICE
xx@desk:~/reproducer$ sudo cp -a reproducer.service /etc/systemd/system/
xx@desk:~/reproducer$ sudo systemctl daemon-reload 
xx@desk:~/reproducer$ sudo systemctl start reproducer.service 
xx@desk:~/reproducer$ ls -lh /proc/$(pgrep -f Reproducer.java)/root
ls: cannot read symbolic link '/proc/2322887/root': Permission denied
lrwxrwxrwx 1 xx xx 0 apr 17 14:43 /proc/2322887/root
xx@desk:~/reproducer$ jcmd $(pgrep -f Reproducer.java) VM.version

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Change the port in the Java source file from 81 to 8081.
Remove AmbientCapabilities=CAP_NET_BIND_SERVICE from /etc/systemd/system/reproducer.service.

xx@desk:~/reproducer$ sudo systemctl daemon-reload 
xx@desk:~/reproducer$ sudo systemctl restart reproducer.service 
xx@desk:~/reproducer$ jcmd $(pgrep -f Reproducer.java) VM.version
2327242:
OpenJDK 64-Bit Server VM version 17.0.6+10-Ubuntu-0ubuntu122.04
JDK 17.0.6
ACTUAL -
xx@desk:~/reproducer$ jcmd $(pgrep -f Reproducer.java) VM.version
2322887:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /proc/2322887/root/tmp/.java_pid2322887: target process 2322887 doesn't respond within 10500ms or HotSpot VM not loaded
	at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:104)
	at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
	at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
	at jdk.jcmd/sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:113)
	at jdk.jcmd/sun.tools.jcmd.JCmd.main(JCmd.java:97)

---------- BEGIN SOURCE ----------
See above.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Not acceptable in all environments, but running jcmd/jstack as root works.

Change the port in the Java source file back to 81.
Enable AmbientCapabilities=CAP_NET_BIND_SERVICE in /etc/systemd/system/reproducer.service.

xx@desk:~/reproducer$ sudo systemctl daemon-reload 
xx@desk:~/reproducer$ sudo systemctl restart reproducer.service 
xx@desk:~/reproducer$ sudo jcmd $(pgrep -f Reproducer.java) VM.version
2328529:
OpenJDK 64-Bit Server VM version 17.0.6+10-Ubuntu-0ubuntu122.04
JDK 17.0.6

FREQUENCY : always



Comments
Duplicate of JDK-8226919
12-02-2024

Additional Information from submitter: ============================== Since I reported this issue, I also found https://bugs.openjdk.org/browse/JDK-8226919 that seems to describe the same underlying problem. I also started digging through the source code to figure out if it's easy to fix and tried to initiate a discussion about it on the serviceability-dev mailing list: https://mail.openjdk.org/pipermail/serviceability-dev/2023-May/048293.html. I would be happy to try to contribute a fix for the issue too if someone is able to guide me through the process.
16-05-2023

Moving it to JDK for analysis.
12-05-2023