JDK-8253714 : [cgroups v2] Soft memory limit incorrectly using memory.high
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 15,16
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: generic
  • Submitted: 2020-09-28
  • Updated: 2022-12-07
  • Resolved: 2020-09-29
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 15 JDK 16 JDK 8 Other
11.0.16-oracleFixed 15.0.2Fixed 16 b18Fixed 8u361Fixed openjdk8u372Fixed
Related Reports
Relates :  
JDK-8230305 - Cgroups v2: Container awareness
Relates :  
JDK-8253727 - [cgroups v2] Memory and swap limits reported incorrectly
Description
As per [1] and [2] the crun OCI runtime supporting cgroups v2 maps --memory-reservation to memory.low. The OpenJDK code still uses memory.high which was a bug in crun fixed in crun 0.11 and better:

$ grep -rn memory\.high src/hotspot/os/linux/
src/hotspot/os/linux/cgroupV2Subsystem_linux.cpp:152:  GET_CONTAINER_INFO_CPTR(cptr, _unified, "/memory.high",
$ grep -rn memory\.high src/java.base/linux/classes/jdk/internal/platform/
src/java.base/linux/classes/jdk/internal/platform/cgroupv2/CgroupV2Subsystem.java:290:        String softLimitStr = CgroupSubsystemController.getStringValue(unified, "memory.high");

This manifests in failing container tests on systems supporting cgroup v2 and having the fixed crun runtime, like Fedora 32.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1880452
[2] https://github.com/containers/crun/blob/master/crun.1.md#cgroup-v2
Comments
Fix Request (8u) Please consider this backport for OpenJDK 8u (targetting 8u372) as part of an effort to backport Cgroups V2 support. This patch is one identified as being good candidates for initial integration (8230305, 8231111, 253714, 8253727, 8224506). Thanks!
15-11-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk8u-dev/pull/130 Date: 2022-10-05 08:41:27 +0000
05-10-2022
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk11u-dev/pull/941 Date: 2022-03-23 13:23:17 +0000
23-03-2022
Fix Request (11u) A clean backport for Cgroup V2 support in 11u
22-03-2022
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk11u-dev/pull/865 Date: 2022-03-08 18:58:11 +0000
08-03-2022
Fix Request (15u): Please approve backporting this fix to JDK 15u. It was the initial release with cgroups v2 support and the bug is reproducible there too. Risk is low as relevant code paths are only executed on cgroup v2 systems. For them it actually fixes the bug if run in a container. The JDK 16 patch applies cleanly to JDK 15u. Testing: container tests on cgroup v2 on Linux x86_64.
01-10-2020
Changeset: ff6843ca Author: Severin Gehwolf <sgehwolf@openjdk.org> Date: 2020-09-29 15:15:50 +0000 URL: https://git.openjdk.java.net/jdk/commit/ff6843ca
29-09-2020
Before: $ podman run --rm -v $(pwd)/build/linux-x86_64-server-release/images/jdk:/jdk -ti --memory-reservation=500m fedora:31 /bin/bash [root@72cff72bb208 /]# /jdk/bin/java -XshowSettings:system -version Operating System Metrics: Provider: cgroupv2 Effective CPU Count: 4 CPU Period: 100000us CPU Quota: -1 CPU Shares: -1 List of Processors: N/A List of Effective Processors: N/A List of Memory Nodes: N/A List of Available Memory Nodes: N/A Memory Limit: Unlimited Memory Soft Limit: Unlimited Memory & Swap Limit: Unlimited openjdk version "16-internal" 2021-03-16 OpenJDK Runtime Environment (build 16-internal+0-adhoc.sgehwolf.jdk-jdk) OpenJDK 64-Bit Server VM (build 16-internal+0-adhoc.sgehwolf.jdk-jdk, mixed mode, sharing) After: $ podman run --rm -v $(pwd)/build/linux-x86_64-server-release/images/jdk:/jdk -ti --memory-reservation=500m fedora:31 /bin/bash [root@4d49147fbc59 /]# /jdk/bin/java -XshowSettings:system -version Operating System Metrics: Provider: cgroupv2 Effective CPU Count: 4 CPU Period: 100000us CPU Quota: -1 CPU Shares: -1 List of Processors: N/A List of Effective Processors: N/A List of Memory Nodes: N/A List of Available Memory Nodes: N/A Memory Limit: Unlimited Memory Soft Limit: 500.00M Memory & Swap Limit: Unlimited openjdk version "16-internal" 2021-03-16 OpenJDK Runtime Environment (build 16-internal+0-adhoc.sgehwolf.jdk-jdk) OpenJDK 64-Bit Server VM (build 16-internal+0-adhoc.sgehwolf.jdk-jdk, mixed mode, sharing
28-09-2020
JDK-8253727 is another fix which is needed for latest cgroups v2 runtimes.
28-09-2020
From the podman-run man page: --memory-reservation=number[unit] Memory soft limit. A unit can be b (bytes), k (kilobytes), m (megabytes), or g (gigabytes). After setting memory reservation, when the system detects memory contention or low memory, containers are forced to restrict their consumption to their reservation. So you should always set the value below --memory, otherwise the hard limit will take precedence. By default, memory reservation will be the same as memory limit.
28-09-2020
On affected systems, test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java, fails with: java.lang.RuntimeException: 'Memory Soft Limit.*524288000' missing from stdout/stderr at jdk.test.lib.process.OutputAnalyzer.shouldMatch(OutputAnalyzer.java:308) at TestMemoryAwareness.testMemorySoftLimit(TestMemoryAwareness.java:108) at TestMemoryAwareness.main(TestMemoryAwareness.java:60) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127) at java.base/java.lang.Thread.run(Thread.java:832)
28-09-2020