JDK-8253727 : [cgroups v2] Memory and swap limits reported incorrectly
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 15,16
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: generic
  • Submitted: 2020-09-28
  • Updated: 2022-12-07
  • Resolved: 2020-10-01
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 15 JDK 16 JDK 8 Other
11.0.16-oracleFixed 15.0.2Fixed 16 b19Fixed 8u361Fixed openjdk8u372Fixed
Related Reports
Relates :  
JDK-8230305 - Cgroups v2: Container awareness
Relates :  
JDK-8253714 - [cgroups v2] Soft memory limit incorrectly using memory.high
Description
The cgroup v2 interface file for swap limits is memory.swap.max. However, in contrast to the cgroup v1 interface file memory.memsw.limit_in_bytes, this file contains only swap limits. The JDKs expectations were written with cgroup v1 interface files in mind.

For example Metrics.getMemoryAndSwapLimit() has this javadoc:

"""
Returns the maximum amount of physical memory and swap space, in bytes, that can be allocated in the Isolation Group.
"""

Yet, the cgroup v2 code only reads memory.swap.max not taking memory.max into account. The results are incorrect. This was previously hiding since the initial crun code seems to have been setting memory.swap.max to the --memory-swap UI option value directly. It's now setting it to $MEMORY_SWAP-$MEMORY so as to mimic cgroups v1 behaviour.

Example:

$ podman run --rm -v $(pwd)/build/linux-x86_64-server-release/images/jdk:/jdk -ti --memory=200m --memory-swap=1g fedora:31 /bin/bash
[root@ce700ea24fd2 /]# /jdk/bin/java -XshowSettings:system -version
Operating System Metrics:
    Provider: cgroupv2
    Effective CPU Count: 4
    CPU Period: 100000us
    CPU Quota: -1
    CPU Shares: -1
    List of Processors: N/A
    List of Effective Processors: N/A
    List of Memory Nodes: N/A
    List of Available Memory Nodes: N/A
    Memory Limit: 200.00M
    Memory Soft Limit: Unlimited
    Memory & Swap Limit: 824.00M

openjdk version "16-internal" 2021-03-16
OpenJDK Runtime Environment (build 16-internal+0-adhoc.sgehwolf.jdk-jdk)
OpenJDK 64-Bit Server VM (build 16-internal+0-adhoc.sgehwolf.jdk-jdk, mixed mode, sharing)

This also yields a test failure in jdk/internal/platform/docker/TestDockerMemoryMetrics.java:

 stdout: [[memoryswap, 200m, 1g]
Exception in thread "main" java.lang.RuntimeException: Memory and swap limit not equal, expected : [209715200, 1073741824], got : [209715200, 864026624]
        at MetricsMemoryTester.testMemoryAndSwapLimit(MetricsMemoryTester.java:150)
        at MetricsMemoryTester.main(MetricsMemoryTester.java:40)
Comments
Fix Request (8u) Please consider this backport for OpenJDK 8u (targetting 8u372) as part of an effort to backport Cgroups V2 support. This patch is one identified as being good candidates for initial integration (8230305, 8231111, 253714, 8253727, 8224506). Thanks!
15-11-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk8u-dev/pull/168 Date: 2022-11-10 16:09:34 +0000
10-11-2022
[~ibereziuk] I see this bug having an jdk8u-ibereziuk label. Are you planning to propose this backport as a PR here: https://github.com/openjdk/jdk8u-dev/?
04-11-2022
Fix Request (OpenJDK 11u): Please approve backporting this to 11u. It's part of the main cgroups v2 support batch. See JDK-8230305 for the list of bugs which should get approved together. Patch applies clean after JDK-8231111 and JDK-8230305 is integrated. Tested manually on a cgroups v2 system. Risk should be low as it's new code not yet present in 11u and fixes a bug introduced with the initial integration of JDK-8231111 and JDK-8230305. Tested with container tests on cgroups v1 and cgroups v2 system. I plan to integrate this together with JDK-8231111 JDK-8230305 JDK-8237479 JDK-8253714.
17-03-2022
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk11u-dev/pull/894 Date: 2022-03-15 18:40:58 +0000
15-03-2022
Changeset: 3e96721c Author: Severin Gehwolf <sgehwolf@openjdk.org> Date: 2020-10-01 09:28:40 +0000 URL: https://git.openjdk.java.net/jdk/commit/3e96721c
01-10-2020
Fix Request (15u): Please approve backporting this fix to JDK 15u. It was the initial release with cgroups v2 support and the bug is reproducible there too. Risk is low as relevant code paths are only executed on cgroup v2 systems. For them it actually fixes the bug if run in a container. The JDK 16 patch applies cleanly to JDK 15u. Testing: container tests on cgroup v2 on Linux x86_64.
01-10-2020
Interface file memory.swap.current is affected too. From the docs[1]: """ memory.swap.current A read-only single value file which exists on non-root cgroups. The total amount of swap currently being used by the cgroup and its descendants. """ [1] https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#memory
29-09-2020
With a fixed build we see what we'd expect: $ podman run --rm -v $(pwd)/build/linux-x86_64-server-release/images/jdk:/jdk -ti --memory=200m --memory-swap=1g fedora:31 /bin/bash [root@63f435d6d007 /]# /jdk/bin/java -XshowSettings:system -version Operating System Metrics: Provider: cgroupv2 Effective CPU Count: 4 CPU Period: 100000us CPU Quota: -1 CPU Shares: -1 List of Processors: N/A List of Effective Processors: N/A List of Memory Nodes: N/A List of Available Memory Nodes: N/A Memory Limit: 200.00M Memory Soft Limit: 0.00K Memory & Swap Limit: 1.00G openjdk version "16-internal" 2021-03-16 OpenJDK Runtime Environment (build 16-internal+0-adhoc.sgehwolf.jdk-jdk) OpenJDK 64-Bit Server VM (build 16-internal+0-adhoc.sgehwolf.jdk-jdk, mixed mode, sharing)
28-09-2020
From podman-run man page: --memory-swap=number[unit] A limit value equal to memory plus swap.
28-09-2020
This affects hotspot and java.base Metrics.java code: $ grep -rn memory.swap.max src/hotspot/os/linux/ src/hotspot/os/linux/cgroupV2Subsystem_linux.cpp:166: GET_CONTAINER_INFO_CPTR(cptr, _unified, "/memory.swap.max", $ grep -rn memory.swap.max src/java.base/linux/classes/ src/java.base/linux/classes/jdk/internal/platform/cgroupv2/CgroupV2Subsystem.java:279: String strVal = CgroupSubsystemController.getStringValue(unified, "memory.swap.max");
28-09-2020