Bug ID: JDK-8252359 HotSpot Not Identifying it is Running in a Container

Type: Bug
Component: hotspot
Sub-Component: runtime
Affected Version: 15

Priority: P3
Status: Resolved
Resolution: Fixed
OS: linux
CPU: x86_64

Submitted: 2020-08-25
Updated: 2024-06-04
Resolved: 2020-09-03

JDK 11	JDK 15	JDK 16	JDK 8	Other
11.0.16Fixed	15.0.2Fixed	16 b15Fixed	8u371Fixed	openjdk8u372Fixed

ADDITIONAL SYSTEM INFORMATION :
Docker Desktop 2.3.0.4 for Mac OS.
Docker Desktop 2.3.0.4 for Windows 10

A DESCRIPTION OF THE PROBLEM :
When running the JVM (15-ea, 16-ea) inside a container on Docker Desktop for Windows (non-WLS2) or Mac OS, the JVM cannot correctly identify it is running in a container, and therefore returns incorrect data about available processors.

Docker Desktop (Windows/Mac) ships a lightweight VM based on LinuxKit with kernel 4.x. When running Docker Desktop for Windows with WSL2 support enabled, this issue does not occur.

This issue was not observed with any release between 14 and 11.

More details on the output of cgroups inside the container running on Docker Desktop for Mac OS:

/ # docker run --cpus=1 -ti openjdk:15-jdk-slim cat /proc/self/mountinfo
557 328 0:123 / / rw,relatime master:126 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/JOQDF3QYKG2BTIR65TELMY6SPN:/var/lib/docker/overlay2/l/KTLKUWGZ4YWDDTHG2BIV7DT2X5:/var/lib/docker/overlay2/l/5ZIBLJPA22CVU5KJMTWHEUFOPD:/var/lib/docker/overlay2/l/IXN4HJHDDFBXWLFKL4QA43533J:/var/lib/docker/overlay2/l/TEBUGLOHYX4U675SMGZOGLH3ZU,upperdir=/var/lib/docker/overlay2/83e3e2ec3cd49f355d0be00a6c5550569053676ff6952e943ac3b7f862ec962e/diff,workdir=/var/lib/docker/overlay2/83e3e2ec3cd49f355d0be00a6c5550569053676ff6952e943ac3b7f862ec962e/work
558 557 0:125 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
559 557 0:126 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
560 559 0:127 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
561 557 0:128 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro
562 561 0:129 / /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,mode=755
563 562 0:29 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/cpuset ro,nosuid,nodev,noexec,relatime master:18 - cgroup cpuset rw,cpuset
564 562 0:30 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/cpu ro,nosuid,nodev,noexec,relatime master:19 - cgroup cpu rw,cpu
565 562 0:31 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/cpuacct ro,nosuid,nodev,noexec,relatime master:20 - cgroup cpuacct rw,cpuacct
566 562 0:32 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/blkio ro,nosuid,nodev,noexec,relatime master:21 - cgroup blkio rw,blkio
567 562 0:33 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/memory ro,nosuid,nodev,noexec,relatime master:22 - cgroup memory rw,memory
568 562 0:34 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/devices ro,nosuid,nodev,noexec,relatime master:23 - cgroup devices rw,devices
569 562 0:35 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/freezer ro,nosuid,nodev,noexec,relatime master:24 - cgroup freezer rw,freezer
570 562 0:36 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/net_cls ro,nosuid,nodev,noexec,relatime master:25 - cgroup net_cls rw,net_cls
571 562 0:37 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/perf_event ro,nosuid,nodev,noexec,relatime master:26 - cgroup perf_event rw,perf_event
572 562 0:38 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/net_prio ro,nosuid,nodev,noexec,relatime master:27 - cgroup net_prio rw,net_prio
573 562 0:39 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/hugetlb ro,nosuid,nodev,noexec,relatime master:28 - cgroup hugetlb rw,hugetlb
574 562 0:40 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/pids ro,nosuid,nodev,noexec,relatime master:29 - cgroup pids rw,pids
575 562 0:41 / /sys/fs/cgroup/rdma ro,nosuid,nodev,noexec,relatime master:30 - cgroup rdma rw,rdma
576 562 0:42 /docker/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63 /sys/fs/cgroup/systemd ro,nosuid,nodev,noexec,relatime master:31 - cgroup cgroup rw,name=systemd
577 559 0:124 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw
578 559 0:130 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k
579 557 254:1 /docker/containers/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/vda1 rw
580 557 254:1 /docker/containers/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63/hostname /etc/hostname rw,relatime - ext4 /dev/vda1 rw
581 557 254:1 /docker/containers/b7dabdf30d0a4dab49b924ee497d83a4dcf008f9cf9423b085be0f9fced35f63/hosts /etc/hosts rw,relatime - ext4 /dev/vda1 rw
329 559 0:127 /0 /dev/console rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
330 558 0:125 /bus /proc/bus ro,relatime - proc proc rw
331 558 0:125 /fs /proc/fs ro,relatime - proc proc rw
332 558 0:125 /irq /proc/irq ro,relatime - proc proc rw
333 558 0:125 /sys /proc/sys ro,relatime - proc proc rw
334 558 0:125 /sysrq-trigger /proc/sysrq-trigger ro,relatime - proc proc rw
357 558 0:131 / /proc/acpi ro,relatime - tmpfs tmpfs ro
358 558 0:126 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
359 558 0:126 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
360 558 0:126 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
361 558 0:126 /null /proc/sched_debug rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
362 561 0:132 / /sys/firmware ro,relatime - tmpfs tmpfs ro

/ # docker run --cpus=1 -ti openjdk:15-jdk-slim cat /proc/cgroups
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	1	28	1
cpu	2	28	1
cpuacct	3	28	1
blkio	4	28	1
memory	5	32	1
devices	6	28	1
freezer	7	28	1
net_cls	8	28	1
perf_event	9	28	1
net_prio	10	28	1
hugetlb	11	28	1
pids	12	28	1
rdma	13	4	1


/ # docker run --cpus=1 -ti openjdk:15-jdk-slim cat /proc/self/cgroup
14:name=systemd:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
13:rdma:/
12:pids:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
11:hugetlb:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
10:net_prio:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
9:perf_event:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
8:net_cls:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
7:freezer:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
6:devices:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
5:memory:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
4:blkio:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
3:cpuacct:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
2:cpu:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975
1:cpuset:/docker/4e650b036993e443ded0122be926a3766a731fdd51d08520bf307c1006219975




Discussion: https://mail.openjdk.java.net/pipermail/discuss/2020-August/005580.html

REGRESSION : Last worked in version 14.0.2

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Install Docker Desktop for Windows, ensure Hyper-V is enabled on Windows, and that WSL2 support is disabled in Docker Desktop preferences.

Install Docker Desktop for Mac OS.

Run the following command:

$ docker run --cpus=1 --memory=256m -ti openjdk:15-jdk-slim java -XX:+PrintFlagsFinal -version 2>&1 | grep Use | grep GC

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
/ # docker run --cpus=1 --memory=256m -ti openjdk:14-jdk-slim java -XX:+PrintFlagsFinal -version 2>&1 | grep Use | grep GC
     bool UseAdaptiveGCBoundary                    = false                                     {product} {default}
     bool UseAdaptiveSizeDecayMajorGCCost          = true                                      {product} {default}
     bool UseAdaptiveSizePolicyWithSystemGC        = false                                     {product} {default}
     bool UseDynamicNumberOfGCThreads              = true                                      {product} {default}
     bool UseG1GC                                  = false                                     {product} {default}
     bool UseGCOverheadLimit                       = true                                      {product} {default}
     bool UseMaximumCompactionOnSystemGC           = true                                      {product} {default}
     bool UseParallelGC                            = false                                     {product} {default}
     bool UseParallelOldGC                         = false                                     {product} {default}
     bool UseSerialGC                              = true                                      {product} {ergonomic}
ACTUAL -
/ # docker run --cpus=1 --memory=256m -ti openjdk:15-jdk-slim java -XX:+PrintFlagsFinal -version 2>&1 | grep Use | grep GC
     bool UseAdaptiveSizeDecayMajorGCCost          = true                                      {product} {default}
     bool UseAdaptiveSizePolicyWithSystemGC        = false                                     {product} {default}
     bool UseDynamicNumberOfGCThreads              = true                                      {product} {default}
     bool UseG1GC                                  = true                                      {product} {ergonomic}
     bool UseGCOverheadLimit                       = true                                      {product} {default}
     bool UseMaximumCompactionOnSystemGC           = true                                      {product} {default}
     bool UseParallelGC                            = false                                     {product} {default}
     bool UseSerialGC                              = false                                     {product} {default}
     bool UseShenandoahGC                          = false                                     {product} {default}
     bool UseZGC                                   = false                                     {product} {default}

CUSTOMER SUBMITTED WORKAROUND :
Use the flag ActiveProcessorCount to force the count to 1.

FREQUENCY : always

Fix request (8u) Please consider this for 8u as part of cgroups v2 support. This was a clean cherry-pick. Thanks!
15-12-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk8u-dev/pull/157 Date: 2022-11-07 10:03:22 +0000
07-11-2022
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk11u-dev/pull/986 Date: 2022-04-01 14:44:42 +0000
01-04-2022
Fix Request (11u) Please consider this for 11u as part of an effort to backport crgoups v2 support. The patch applies clean and the test passes in a pure v2 environment. Thanks!
01-04-2022
Fix Request (15u): Please approve backporting this to JDK 15u as this fixes a regression of container detection on some systems. It doesn't affect JDK 14 and earlier as those don't have the cgroups v2 patch. The JDK 16 patch applies cleanly and passes the changed regression test. The risk should be OK (only affects container detection code path which can be turned off with -XX:-UseContainerSupport). Linux-only change. Testing showed, that existing cgroupv1 and cgroupv2 support continues to be working with this.
16-09-2020
URL: https://hg.openjdk.java.net/jdk/jdk/rev/0178d0f136e9 User: bobv Date: 2020-09-03 15:25:25 +0000
03-09-2020
I've now also added a test for the Java side of things: https://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8252359/02/webrev/ As it's passing it seems to suggest Metrics.java isn't affected by this. We need to confirm with the original reporter.
02-09-2020
[~bobv] I've crafted a regression test for this here. Feel free to use it: https://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8252359/01/webrev/ Fails for unpatched with: java.lang.RuntimeException: Windows/Mac container engine should be cgroupsv1: expected 1 to equal 4 at jdk.test.lib.Asserts.fail(Asserts.java:594) at jdk.test.lib.Asserts.assertEquals(Asserts.java:205) at jdk.test.lib.Asserts.assertEQ(Asserts.java:178) at CgroupSubsystemFactory.testCgroupv1WinMac(CgroupSubsystemFactory.java:263) at CgroupSubsystemFactory.main(CgroupSubsystemFactory.java:334) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127) at java.base/java.lang.Thread.run(Thread.java:832) Passes with your candidate fix, but fails testCgroupv2NoCgroup2Fs() test.
02-09-2020
ILW = HLM = P3
01-09-2020
It would be ideal to write a test case for this issue first. We have a couple of tests with various combinations of /proc/self/mountinfo files and resulting outcomes for container detection: test/jdk/jdk/internal/platform/cgroup/TestCgroupSubsystemFactory.java test/hotspot/jtreg/containers/cgroup/CgroupSubsystemFactory.java This should avoid regressing this again. One issue I remember running into on cgroups v2 was when short matches for files the likes of the following were encountered: 31 30 0:27 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:5 - cgroup2 cgroup2 rw,seclabel,nsdelegate
01-09-2020
The problem seems to be caused by our improper scanning of /proc/mountinfo. Failing System 439 434 0:33 /docker/0a67832faae434cc8fa0a942e04a308bb8b120b5099018f1ee09f1d7a52746b7 /sys/fs/cgroup/memory ro,nosuid,nodev,noexec,relatime master:22 - cgroup memory rw,memory Working System 36 31 0:31 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,memory When the cgroupv2 support was added we started scanning the second “cgroup” entry in /proc/mountinfo as the file system type. According to the man pages, the first “cgroup” is the file system type. Entry position 10 in [1]mountinfo is “mount source: filesystem-specific information”. All systems we’ve tested on thus far have had two cgroup strings so this regression was not caught. [1] https://man7.org/linux/man-pages/man5/proc.5.htm
01-09-2020
OS: windows 10 Downloaded Docker desktop for windows version 2.3.0.4 Using Windows powershell Could not get all the parameters mentioned in the reproducer description. Below are the details: For OpenJDK 15 => docker run --cpus=1 --memory=256m -ti openjdk:15-jdk java -XX:+PrintFlagsFinal -version 2>&1 \| Select-String 'Use' \| Select-String 'GC' Output: uintx GCPauseIntervalMillis = 0 UseMaximumCompactionOnSystemGC = true bool UseZGC = false For OpenJDK 14=> docker run --cpus=1 --memory=256m -ti openjdk:14-jdk java -XX:+PrintFlagsFinal -version 2>&1 \| Select-String 'Use'\| Select-String 'GC' Output: uintx GCPauseIntervalMillis = 0 bool UseSerialGC = true Moving it to dev team for further analysis.
26-08-2020