JDK-8253435 : Cgroup: 'stomping of _mount_path' crash if manually mounted cpusets exist
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 15
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: generic
  • Submitted: 2020-09-21
  • Updated: 2023-01-12
  • Resolved: 2020-09-28
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 16 JDK 8 Other
11.0.16Fixed 16 b18Fixed 8u371Fixed openjdk8u372Fixed
Related Reports
Relates :  
JDK-8253939 - [TESTBUG] Increase coverage of the cgroups detection code
Relates :  
JDK-8270087 - Manually mounted cgroups controllers introduce warning logs on stdout
Relates :  
JDK-8239785 - Cgroups: Incorrect detection logic on old systems in hotspot
Description
If a tool like cpuset (https://github.com/lpechacek/cpuset) is used to manually create and manage cpusets (https://man7.org/linux/man-pages/man7/cpuset.7.html) the cgroups detections will be confused and crash in a debug build (see below) or behave unexpectedly in a product build.

The problem is that the additionally mounted cpuset will be interpreted as if it was belonging to Cgroup controller:

$ grep cgroup /proc/self/mountinfo 
36 25 0:30 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755
49 36 0:43 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:23 - cgroup cgroup rw,memory
50 36 0:44 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:24 - cgroup cgroup rw,rdma
...
43 36 0:37 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,cpuset
121 32 0:37 / /cpusets rw,relatime shared:69 - cgroup none rw,cpuset

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/priv/simonisv/OpenJDK/Git/jdk/src/hotspot/os/linux/cgroupSubsystem_linux.cpp:305), pid=1583, tid=1601
#  assert(cg_infos[0]._mount_path == __null) failed: stomping of _mount_path
#
# JRE version:  (16.0) (slowdebug build )
# Java VM: OpenJDK 64-Bit Server VM (slowdebug 16-internal+0-adhoc.simonisv.jdk, mixed mode, sharing, tiered, unknown gc, linux-amd64)
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
#

---------------  S U M M A R Y ------------

Command Line: 

Host: xxx, Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 8 cores, 31G, Ubuntu 18.04.5 LTS
Time: Mon Sep 21 17:35:18 2020 CEST elapsed time: 0.006970 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread is native thread

Stack: [0x00007ffff7ed0000,0x00007ffff7fd1000],  sp=0x00007ffff7fc92a0,  free space=996k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1217bf8]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x7aa
V  [libjvm.so+0x12173eb]  VMError::report_and_die(Thread*, void*, char const*, int, char const*, char const*, __va_list_tag*)+0x57
V  [libjvm.so+0x74bda5]  report_vm_error(char const*, int, char const*, char const*, ...)+0x205
V  [libjvm.so+0x620736]  CgroupSubsystemFactory::determine_type(CgroupInfo*, char const*, char const*, char const*, unsigned char*)+0xc62
V  [libjvm.so+0x61f5fb]  CgroupSubsystemFactory::create()+0xbf
V  [libjvm.so+0xeeccb3]  OSContainer::init()+0xb7
V  [libjvm.so+0xef9275]  os::pd_init_container_support()+0x9
V  [libjvm.so+0x464e65]  os::init_container_support()+0x9
V  [libjvm.so+0x45e095]  Arguments::parse_vm_init_args(JavaVMInitArgs const*, JavaVMInitArgs const*, JavaVMInitArgs const*, JavaVMInitArgs const*)+0x147
V  [libjvm.so+0x4628e8]  Arguments::parse(JavaVMInitArgs const*)+0x3fe
V  [libjvm.so+0x1191a69]  Threads::create_vm(JavaVMInitArgs*, bool*)+0xb3
V  [libjvm.so+0xade81d]  JNI_CreateJavaVM_inner(JavaVM_**, void**, void*)+0xda
V  [libjvm.so+0xadeb25]  JNI_CreateJavaVM+0x32
C  [libjli.so+0x735a]  InitializeJVM+0x13a
C  [libjli.so+0x3f3c]  JavaMain+0xdc
C  [libjli.so+0xaca2]  ThreadJavaMain+0x27
Comments
Fix Request (8u) Please consider this backport to 8u for cgroups v2 support. It wasn't clean, needed logging changes and test fix-ups. Thanks
15-12-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk8u-dev/pull/158 Date: 2022-11-07 10:22:13 +0000
07-11-2022
The actual issue will be addressed with JDK-8293472
07-09-2022
That warning potentially indicates a problem, so lowering it some other level, would a) not fix it (if info), b) risk that the problem would go unnoticed. Neither are good options. As indicated on the gradle issue, -XX:-UseContainerSupport would be a work-around.
31-08-2022
The new warnings emitted by this change[1][2] can cause problems for Gradle: https://github.com/gradle/gradle/issues/21616 Maybe the log level could be made lower for these messages? [1] https://github.com/openjdk/jdk/blob/f694f8a7671002559e7d23fdb65d5e9c768f9c03/src/hotspot/os/linux/cgroupSubsystem_linux.cpp#L346 [2] https://github.com/openjdk/jdk/blob/f694f8a7671002559e7d23fdb65d5e9c768f9c03/src/hotspot/os/linux/cgroupSubsystem_linux.cpp#L351
31-08-2022
Fix request (11u) Please consider this backport for 11u as part of the cgroups2 backport effort. It's clean and the included test passes on a cgroups v2 system. Thanks!
04-04-2022
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk11u-dev/pull/992 Date: 2022-04-04 09:27:08 +0000
04-04-2022
Changeset: 0054c15f Author: Volker Simonis <simonis@openjdk.org> Date: 2020-09-28 15:42:10 +0000 URL: https://git.openjdk.java.net/jdk/commit/0054c15f
28-09-2020
[~simonis] How does /proc/self/cgroup look like on the system you have with manual /cpusets mounts?
28-09-2020
2:cpuset:/ Is the relevant line. So in your case the path to cgroup v1 interface files for the cset-controlled cgroup controller would be /cpuset. I think that's fine and accounts for the case [~bobv] mentioned in the review. That's why I was asking. I'm not sure whether cset controlled only systems - without mounts at /sys/fs/cgroup - exist.
28-09-2020
I'm on Ubuntu where cpuset is a supported package (see https://launchpad.net/ubuntu/+source/cpuset). I think on OpenSuSE it is supported as well (see https://software.opensuse.org/package/cpuset). When calling cpuset's command line utility "cset" for the first time on Ubuntu 18.04 (e.g. "cset set --list") it mounts cgroup on "/cpusets" as described in the bug report. I don't think it changes "/proc/self/cgroup" which looks as follows: # cat /proc/self/cgroup 12:devices:/user.slice 11:cpu,cpuacct:/user.slice 10:perf_event:/ 9:rdma:/ 8:pids:/user.slice/user-633506465.slice/user@633506465.service 7:hugetlb:/ 6:freezer:/ 5:blkio:/user.slice 4:memory:/user.slice 3:net_cls,net_prio:/ 2:cpuset:/ 1:name=systemd:/user.slice/user-633506465.slice/user@633506465.service/gnome-terminal-server.service 0::/user.slice/user-633506465.slice/user@633506465.service/gnome-terminal-server.service
28-09-2020
ILW = HLM = P3
22-09-2020
Question is whether this would be valid too? I see no reason why it shouldn't be, though. 43 36 0:37 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:17 - cgroup none rw,cpuset 121 32 0:37 / /cpusets rw,relatime shared:69 - cgroup none rw,cpuset
22-09-2020
So the problematic lines are these: 43 36 0:37 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,cpuset 121 32 0:37 / /cpusets rw,relatime shared:69 - cgroup none rw,cpuset I don't know how cpuset[1] works, but it might be pure luck that it's using "none" as mount source. For example in JDK-8252359 we've discovered that the mount source might be one of the controller names. In fact, we've changed some tests to refer to "none" in that bug, which seems to suggest that the proposed patch break the test. [1] https://github.com/lpechacek/cpuset
22-09-2020