JDK-8250984 : Memory Docker tests fail on some Linux kernels w/o cgroupv1 swap limit capabilities
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 15,16
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2020-08-03
  • Updated: 2021-05-13
  • Resolved: 2020-09-25
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 13 JDK 15 JDK 16 JDK 8 Other
11.0.10-oracleFixed 13.0.6Fixed 15.0.4Fixed 16 b18Fixed 8u271Fixed openjdk8u292Fixed
Related Reports
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Sub Tasks
JDK-8250986 :  
Tests that fail:

Failure description:
  stderr: [WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.]

java.lang.RuntimeException: 'OperatingSystemMXBean.getFreeSwapSpaceSize: [1-9][0-9]+' missing from stdout/stderr

Kernel: linux-5.4.0-1019-oracle
Fix request (15u) Requesting backport to 15u for parity with 11u. The patch applies almost cleanly except for changes in ProblemLists, they are skipped as not applicable to 15u. Tested with tier1 and container tests. Follow-up fix JDK-8257746 is planned to be backported as well. RFR: http://mail.openjdk.java.net/pipermail/jdk-updates-dev/2021-May/006145.html

Fix Request (OpenJDK 8u) Please approve this 8u backport (I see this fix in Oracle JDK 8u281). It depends on JDK-8217338 which introduces MemorySubSystem class. The patch didn't apply cleanly due to the jdk/hotspot repository split in 8u. It has been reviewed by Paul Hohensee. webrev: https://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8250984/jdk8/01/ RFR: https://mail.openjdk.java.net/pipermail/jdk8u-dev/2020-November/013034.html Mentioned tests fails on a cgroups v1 system with swapaccount=0 before the patch and passes after.

Fix request (13u) Requesting backport to 13u for parity with 11u. The patch doesn't apply cleanly since 13u doesn't have cgroups v2 support (JDK-8231111), reapplied manually. Tested with tier1 and container tests. RFR approval: http://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-December/004352.html

8u review approval: https://mail.openjdk.java.net/pipermail/jdk8u-dev/2020-November/013039.html

Fix Request (OpenJDK 11u): Please approve this backport for OpenJDK 11u. I see Oracle backported it too. The patch doesn't apply cleanly, since jdk head have the cgroups v2 refactoring. Patch has been reviewed by Aleksey Shipilev. Tested this on a system with swapaccount=0 which fails the test before and passes after. In general, this is a patch which improves resiliency so should be fairly low risk. No change for systems with swap accounting on; reports memory only - without swap - for systems with swap accounting off. Tier 1 tests pass too. webrev: https://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8250984/01/webrev/ RFR: http://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-October/003939.html

Changeset: 01875677 Author: Harold Seigel <hseigel@openjdk.org> Date: 2020-09-25 17:16:38 +0000 URL: https://git.openjdk.java.net/jdk/commit/01875677

What happens when running test jdk/internal/platform/docker/TestDockerMemoryMetrics.java is that method getStringValue(), in jdk/internal/platform/CgroupSubsystemController.java, gets an IO Exception: java.nio.file.NoSuchFileException: /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes, causing it to return null: public static String getStringValue(CgroupSubsystemController controller, String param) { if (controller == null) return null; try { return CgroupUtil.readStringValue(controller, param); } catch (IOException e) { return null; } } This method then returns -1, because strval is null: public static long getLongValue(CgroupSubsystemController controller, String param, Function<String, Long> conversion, long defaultRetval) { String strval = getStringValue(controller, param); if (strval == null) return defaultRetval; return conversion.apply(strval); } Control eventually passes back to getTotalSwapSpaceSize(), in src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java, which sees that limit is -1, so it returns the value for the system swap space, which, in this case, is 0. public long getTotalSwapSpaceSize() { if (containerMetrics != null) { long limit = containerMetrics.getMemoryAndSwapLimit(); // The memory limit metrics is not available if JVM runs on Linux host (not in a docker container) // or if a docker container was started without specifying a memory limit (without '--memory=' // Docker option). In latter case there is no limit on how much memory the container can use and // it can use as much memory as the host's OS allows. long memLimit = containerMetrics.getMemoryLimit(); if (limit >= 0 && memLimit >= 0) { return limit - memLimit; // might potentially be 0 for limit == memLimit } } return getTotalSwapSpaceSize0(); }

Thanks. This seems to suggest that the host/system values are being reported as 0: OperatingSystemMXBean.getTotalSwapSpaceSize: 0 OperatingSystemMXBean.getFreeSwapSpaceSize: 0 That's different to what we have seen with JDK-8244500 in the past :(

Metrics instance: non-null Metrics.getMemoryAndSwapLimit() == -1 Metrics.getMemoryLimit() == 104857600 Metrics.getMemoryAndSwapUsage() == -1 Metrics.getMemoryUsage() == 34107392

There should be some traces about what the Java Metrics system thinks the limits are on such systems: From test/hotspot/jtreg/containers/docker/CheckOperatingSystemMXBean.java: System.out.println("Metrics instance: " + (metrics == null ? "null" : "non-null")); if (metrics != null) { System.out.println("Metrics.getMemoryAndSwapLimit() == " + metrics.getMemoryAndSwapLimit()); System.out.println("Metrics.getMemoryLimit() == " + metrics.getMemoryLimit()); System.out.println("Metrics.getMemoryAndSwapUsage() == " + metrics.getMemoryAndSwapUsage()); System.out.println("Metrics.getMemoryUsage() == " + metrics.getMemoryUsage()); } It would be good to know what it reports. This bug is very similar to JDK-8244500 and we've had several iterations on trying to getting it fixed.

Test containers/docker/TestMemoryAwareness.java is failing with these results: OperatingSystemMXBean.getTotalSwapSpaceSize: 0 OperatingSystemMXBean.getFreeSwapSpaceSize: 0 stderr: [WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.

Runtime.availableProcessors: 4 [0.379s][trace][os,container] CgroupSubsystem::active_processor_count (cached): 4 OperatingSystemMXBean.getAvailableProcessors: 4 [0.388s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us [0.388s][trace][os,container] CPU Quota is: -1 [0.389s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us [0.389s][trace][os,container] CPU Period is: 100000 [0.389s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/cpu.shares OperatingSystemMXBean.getTotalMemorySize: 104857600 [0.389s][trace][os,container] CPU Shares is: 1024 [0.389s][trace][os,container] OSContainer::active_processor_count: 4 OperatingSystemMXBean.getTotalPhysicalMemorySize: 104857600 OperatingSystemMXBean.getFreeMemorySize: 69091328 [0.391s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes [0.391s][trace][os,container] Memory Limit is: 104857600 [0.391s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.391s][trace][os,container] Memory Usage is: 35762176 OperatingSystemMXBean.getFreePhysicalMemorySize: 69091328 [0.392s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.392s][trace][os,container] Memory Usage is: 35758080 OperatingSystemMXBean.getTotalSwapSpaceSize: 0 OperatingSystemMXBean.getFreeSwapSpaceSize: 0 [0.393s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.393s][trace][os,container] Memory Usage is: 35753984 [0.394s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.394s][trace][os,container] Memory Usage is: 35880960 [0.394s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.394s][trace][os,container] Memory Usage is: 35876864 [0.394s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.395s][trace][os,container] Memory Usage is: 35872768 [0.397s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.397s][trace][os,container] Memory Usage is: 35971072 [0.397s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.397s][trace][os,container] Memory Usage is: 35909632 [0.398s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.398s][trace][os,container] Memory Usage is: 35905536 OperatingSystemMXBean.getCpuLoad: 0.000000 [0.401s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.401s][trace][os,container] Memory Usage is: 35897344 OperatingSystemMXBean.getSystemCpuLoad: 0.666667 [0.402s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.402s][trace][os,container] Memory Usage is: 35893248 [0.403s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.403s][trace][os,container] Memory Usage is: 35889152 [0.404s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.404s][trace][os,container] Memory Usage is: 35778560 ]; stderr: [WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. ] exitValue = 0 java.lang.RuntimeException: 'OperatingSystemMXBean\\.getFreeSwapSpaceSize: [1-9][0-9]+' missing from stdout/stderr at jdk.test.lib.process.OutputAnalyzer.shouldMatch(OutputAnalyzer.java:308) at TestMemoryAwareness.testOperatingSystemMXBeanAwareness(TestMemoryAwareness.java:159) at TestMemoryAwareness.main(TestMemoryAwareness.java:66) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298) at java.base/java.lang.Thread.run(Thread.java:832) JavaTest Message: Test threw exception: java.lang.RuntimeException JavaTest Message: shutting down test result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: 'OperatingSystemMXBean\.getFreeSwapSpaceSize: [1-9][0-9]+' missing from stdout/stderr