JDK-8258954 : CGroups - invalid detection of cgroupv2
  • Type: Bug
  • Component: other-libs
  • Affected Version: 15.0.1
  • Priority: P4
  • Status: New
  • Resolution: Unresolved
  • OS: linux_ubuntu
  • CPU: x86_64
  • Submitted: 2020-12-18
  • Updated: 2022-03-22
Related Reports
Relates :  
JDK-8239559 - Cgroups: Incorrect detection logic on some systems
Description
ADDITIONAL SYSTEM INFORMATION :
Content of /proc/self/mountinfo
16 21 0:15 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
17 21 0:3 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
18 21 0:5 / /dev rw,relatime - devtmpfs udev rw,size=8206048k,nr_inodes=2051512,mode=755
19 18 0:12 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
20 21 0:16 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=1643364k,mode=755
21 0 8:5 / / rw,relatime - ext4 /dev/disk/by-uuid/812d6452-9193-4e6d-a9fd-259061b14840 rw,errors=remount-ro,data=ordered
23 16 0:17 / /sys/fs/cgroup rw,relatime - tmpfs none rw,size=4k,mode=755
24 16 0:18 / /sys/fs/fuse/connections rw,relatime - fusectl none rw
25 16 0:6 / /sys/kernel/debug rw,relatime - debugfs none rw
26 16 0:10 / /sys/kernel/security rw,relatime - securityfs none rw
27 20 0:19 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=5120k
28 20 0:20 / /run/shm rw,nosuid,nodev,relatime - tmpfs none rw
29 20 0:21 / /run/user rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=102400k,mode=755
30 16 0:22 / /sys/fs/pstore rw,relatime - pstore none rw
31 21 8:17 / /opt rw,relatime - ext4 /dev/sdb1 rw,data=ordered
33 23 0:24 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime - cgroup systemd rw,name=systemd

content of /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  0       1       1
cpu     0       1       1
cpuacct 0       1       1
memory  0       1       1
devices 0       1       1
freezer 0       1       1
net_cls 0       1       1
blkio   0       1       1
perf_event      0       1       1
net_prio        0       1       1
hugetlb 0       1       1



A DESCRIPTION OF THE PROBLEM :
I receive exception on my linux server

Exception in thread "main" java.lang.InternalError: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:65)
        at java.base/jdk.internal.platform.Container.metrics(Container.java:43)
        at jdk.management/com.sun.management.internal.OperatingSystemImpl.<init>(OperatingSystemImpl.java:48)
        at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean(PlatformMBeanProviderImpl.java:281)
        at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap(PlatformMBeanProviderImpl.java:198)
        at java.management/sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans(PlatformMBeanProvider.java:195)
        at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:686)
        at java.management/java.lang.management.ManagementFactory.getOperatingSystemMXBean(ManagementFactory.java:388)
        at testj15.Main.main(Main.java:9)
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:61)
        ... 8 more
Caused by: java.lang.ExceptionInInitializerError
        at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:77)
        at java.base/jdk.internal.platform.CgroupMetrics.getInstance(CgroupMetrics.java:163)
        ... 13 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 1
        at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.initSubsystem(CgroupV2Subsystem.java:71)
        at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.<clinit>(CgroupV2Subsystem.java:42)
        ... 15 more

After checking for it I found than invalid version of cgroup was detected.
This issue referenced to https://bugs.openjdk.java.net/browse/JDK-8239559

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Create test project with Main-file 
	public static void main(String[] args) throws Exception {
		ManagementFactory.getOperatingSystemMXBean();
	}

2. Create file mountinfo with content
16 21 0:15 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
17 21 0:3 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
18 21 0:5 / /dev rw,relatime - devtmpfs udev rw,size=8206048k,nr_inodes=2051512,mode=755
19 18 0:12 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
20 21 0:16 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=1643364k,mode=755
21 0 8:5 / / rw,relatime - ext4 /dev/disk/by-uuid/812d6452-9193-4e6d-a9fd-259061b14840 rw,errors=remount-ro,data=ordered
23 16 0:17 / /sys/fs/cgroup rw,relatime - tmpfs none rw,size=4k,mode=755
24 16 0:18 / /sys/fs/fuse/connections rw,relatime - fusectl none rw
25 16 0:6 / /sys/kernel/debug rw,relatime - debugfs none rw
26 16 0:10 / /sys/kernel/security rw,relatime - securityfs none rw
27 20 0:19 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=5120k
28 20 0:20 / /run/shm rw,nosuid,nodev,relatime - tmpfs none rw
29 20 0:21 / /run/user rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=102400k,mode=755
30 16 0:22 / /sys/fs/pstore rw,relatime - pstore none rw
31 21 8:17 / /opt rw,relatime - ext4 /dev/sdb1 rw,data=ordered
33 23 0:24 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime - cgroup systemd rw,name=systemd

3. create file cgroup with content
 #subsys_name    hierarchy       num_cgroups     enabled
cpuset  0       1       1
cpu     0       1       1
cpuacct 0       1       1
memory  0       1       1
devices 0       1       1
freezer 0       1       1
net_cls 0       1       1
blkio   0       1       1
perf_event      0       1       1
net_prio        0       1       1
hugetlb 0       1       1

4. Setup 2 breakpoints:
4.1 jdk.internal.platform.CgroupSubsystemFactory:86
Map<String, CgroupInfo> infos = new HashMap<>();

4.2 jdk.internal.platform.cgroupv2.CgroupV2Subsystem:64
        String mountPath = null;

5. Run test project under debug.
When debugger stopped on breakpoint 4.1, replace variable values:
5.1 write file path of mountinfo, created on step 2 to variable
String mountInfo - jdk.internal.platform.CgroupSubsystemFactory.determineType(String, String)

5.2 write file path of cgroup, created on step 3 to variable 
String cgroups - jdk.internal.platform.CgroupSubsystemFactory.determineType(String, String)

6. Continue program execution. It will stopped on second break point.

7. Press 2-3 times Step-Into until you will enter to java.nio.file.Paths:69

8. Replace variable value:
write file path of mountinfo, created on step 2 to variable
String first - java.nio.file.Paths.get(String first, String... more)

9. Continue program execution.


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Program executed successfully.
ACTUAL -
Raised exception with stack trace

Exception in thread "main" java.lang.InternalError: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:65)
	at java.base/jdk.internal.platform.Container.metrics(Container.java:43)
	at jdk.management/com.sun.management.internal.OperatingSystemImpl.<init>(OperatingSystemImpl.java:48)
	at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean(PlatformMBeanProviderImpl.java:281)
	at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap(PlatformMBeanProviderImpl.java:198)
	at java.management/sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans(PlatformMBeanProvider.java:195)
	at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:686)
	at java.management/java.lang.management.ManagementFactory.getOperatingSystemMXBean(ManagementFactory.java:388)
	at testj15.Main.main(Main.java:7)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:61)
	... 8 more
Caused by: java.lang.ExceptionInInitializerError
	at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:77)
	at java.base/jdk.internal.platform.CgroupMetrics.getInstance(CgroupMetrics.java:163)
	... 13 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 1
	at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.initSubsystem(CgroupV2Subsystem.java:71)
	at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.<clinit>(CgroupV2Subsystem.java:42)
	... 15 more


---------- BEGIN SOURCE ----------
package testj15;

import java.lang.management.ManagementFactory;

public class Main {
	public static void main(String[] args) throws Exception {
		ManagementFactory.getOperatingSystemMXBean();
	}
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
No workaround found at this moment

FREQUENCY : always
Comments
Is this still reproducible with JDK 17 or 18? Looks to me like a duplicate of JDK-8245543
22-03-2022
The observations on Oracle Linux: JDK 16: Not applicable JDK 15: Failed, java.lang.reflect.InvocationTargetException was thrown JDK 14: Not applicable
26-12-2020
Additional information from the submitter: I re-read proposed fix and found error try (Stream<String> mntInfo = CgroupUtil.readFilePrivileged(Paths.get(mountInfo))) { boolean anyCgroupMounted = mntInfo.anyMatch(line -> line.contains("cgroup")); if (!anyCgroupMounted && anyCgroupsV1Controller) { return Optional.empty(); } anyCgroupMounted = mntInfo.anyMatch(line -> line.contains(" - cgroup2 ")); if (!anyCgroupMounted && anyCgroupsV2Controller) { return Optional.empty(); } } is incorrect. Sorry, I cant check code -I have no openjdk sources locally. This must be look like try (Stream<String> mntInfo = CgroupUtil.readFilePrivileged(Paths.get(mountInfo))) { boolean anyCgroupMounted = mntInfo.anyMatch(line -> line.contains("cgroup")); if (!anyCgroupMounted && anyCgroupsV1Controller) { return Optional.empty(); } } try (Stream<String> mntInfo = CgroupUtil.readFilePrivileged(Paths.get(mountInfo))) { boolean anyCgroupMounted = mntInfo.anyMatch(line -> line.contains(" - cgroup2 ")); if (!anyCgroupMounted && anyCgroupsV2Controller) { return Optional.empty(); } }
26-12-2020
Additional information from the submitter: For a starts - reproducing. Method Optional<CgroupTypeResult> jdk.internal.platform.CgroupSubsystemFactory.determineType(String mountInfo, String cgroups) throws IOException Is very simple and have no side-effects when correctly files passed. I attached files I used for test - please, try it. Second - bug reasons and workaround. I use ubuntu. This reproduced on 3 build-servers only with ubuntu 14-04 I think this related to init.d and systemd. This server used init.d at first for a startup and then it was upgrade to systemd. I think cgroups was not properly initialized during this process. At now it looks like: root@build00:/home/d.shmyglev# ls -la /sys/fs/cgroup/ итого 0 drwxr-xr-x 3 root root 60 дек. 23 10:17 . drwxr-xr-x 7 root root 0 нояб. 22 2018 .. dr-xr-xr-x 3 root root 0 сент. 26 2018 systemd root@build00:/home/d.shmyglev# mount | grep cgroup none on /sys/fs/cgroup type tmpfs (rw) systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd) (systemd only mounted, no any other!) Ok, lets look in JAVA. jdk.internal.platform.CgroupSubsystemFactory used very simple algorithm for a check which subsystem enabled: cgroup, cgroupv2 or both. It reads /proc/cgroups and checks hierarchy field equals zero. If all rows have hierarchy == 0 - than it think that cgroupv2 used. Next, it reads /proc/self/mountinfo for a any row, contains 'cgroup' substring. This check passed betcause systemd cgroup mounted.. But this check is incorrect! This is first error. For cgroupv2 it must look for a 'cgroupv2' substring! Next. We go into jdk.internal.platform.cgroupv2.CgroupV2Subsystem.initSubsystem() It checked /proc/self/mountinfo for a ' - cgroup2 ' substring. This check differences from early passed (This is second error) and found no lines. Collectors.joining() returned empty string, which splited by space to a array with size == 1. Than code take 4-th token from array without array-size-check. This is third error. Java raises java.lang.ArrayIndexOutOfBoundsException. Proposed fix: 1. jdk.internal.platform.CgroupSubsystemFactory:121 This block have to be changed from try (Stream<String> mntInfo = CgroupUtil.readFilePrivileged(Paths.get(mountInfo))) { boolean anyCgroupMounted = mntInfo.anyMatch(line -> line.contains("cgroup")); if (!anyCgroupMounted && isCgroupsV2) { return Optional.empty(); } } to somethink like try (Stream<String> mntInfo = CgroupUtil.readFilePrivileged(Paths.get(mountInfo))) { boolean anyCgroupMounted = mntInfo.anyMatch(line -> line.contains("cgroup")); if (!anyCgroupMounted && anyCgroupsV1Controller) { return Optional.empty(); } anyCgroupMounted = mntInfo.anyMatch(line -> line.contains(" - cgroup2 ")); if (!anyCgroupMounted && anyCgroupsV2Controller) { return Optional.empty(); } } 2. jdk.internal.platform.cgroupv2.CgroupV2Subsystem:70-71 There must be added lines to check tokens[] size. if (tokens.length < 4) { return null; } Workaround: We can trigger check jdk.internal.platform.CgroupSubsystemFactory:70 by using 2 simple commands in console (1.jar - compiled sample) 1. Error reproduced: root@build00:/home/d.shmyglev# /opt/java15/bin/java --version openjdk 15.0.1 2020-10-20 OpenJDK Runtime Environment (build 15.0.1+9-18) OpenJDK 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing) root@build00:/home/d.shmyglev# /opt/java15/bin/java -cp 1.jar testj15.Main Exception in thread "main" java.lang.InternalError: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:65) at java.base/jdk.internal.platform.Container.metrics(Container.java:43) at jdk.management/com.sun.management.internal.OperatingSystemImpl.<init>(OperatingSystemImpl.java:48) at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean(PlatformMBeanProviderImpl.java:281) at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap(PlatformMBeanProviderImpl.java:198) at java.management/sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans(PlatformMBeanProvider.java:195) at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:686) at java.management/java.lang.management.ManagementFactory.getOperatingSystemMXBean(ManagementFactory.java:388) at testj15.Main.main(Main.java:7) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:61) ... 8 more Caused by: java.lang.ExceptionInInitializerError at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:77) at java.base/jdk.internal.platform.CgroupMetrics.getInstance(CgroupMetrics.java:163) ... 13 more Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 1 at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.initSubsystem(CgroupV2Subsystem.java:71) at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.<clinit>(CgroupV2Subsystem.java:42) ... 15 more 2. fix cgroup initialization. root@build00:/home/d.shmyglev# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 0 1 1 cpu 0 1 1 cpuacct 0 1 1 memory 0 1 1 devices 0 1 1 freezer 0 1 1 net_cls 0 1 1 blkio 0 1 1 perf_event 0 1 1 net_prio 0 1 1 hugetlb 0 1 1 root@build00:/home/d.shmyglev# mkdir /sys/fs/cgroup/cpuset root@build00:/home/d.shmyglev# mount -t cgroup cpuset -o cpuset /sys/fs/cgroup/cpuset/ root@build00:/home/d.shmyglev# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 4 1 1 cpu 0 1 1 cpuacct 0 1 1 memory 0 1 1 devices 0 1 1 freezer 0 1 1 net_cls 0 1 1 blkio 0 1 1 perf_event 0 1 1 net_prio 0 1 1 hugetlb 0 1 1 After this one string appeared in /proc/cgroups with hierarchy <> 0 3. Check it: root@build00:/home/d.shmyglev# /opt/java15/bin/java -cp 1.jar testj15.Main root@build00:/home/d.shmyglev# ` all ok!
26-12-2020
Requested the clarification of the results from the submitter.
26-12-2020