Bug ID: JDK-8250984 Memory Docker tests fail on some Linux kernels w/o cgroupv1 swap limit capabilities

JDK 11	JDK 13	JDK 15	JDK 16	JDK 8	Other
11.0.10-oracleFixed	13.0.6Fixed	15.0.4Fixed	16 b18Fixed	8u271Fixed	openjdk8u292Fixed

Fix request (15u) Requesting backport to 15u for parity with 11u. The patch applies almost cleanly except for changes in ProblemLists, they are skipped as not applicable to 15u. Tested with tier1 and container tests. Follow-up fix JDK-8257746 is planned to be backported as well. RFR: http://mail.openjdk.java.net/pipermail/jdk-updates-dev/2021-May/006145.html
11-05-2021
Fix Request (OpenJDK 8u) Please approve this 8u backport (I see this fix in Oracle JDK 8u281). It depends on JDK-8217338 which introduces MemorySubSystem class. The patch didn't apply cleanly due to the jdk/hotspot repository split in 8u. It has been reviewed by Paul Hohensee. webrev: https://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8250984/jdk8/01/ RFR: https://mail.openjdk.java.net/pipermail/jdk8u-dev/2020-November/013034.html Mentioned tests fails on a cgroups v1 system with swapaccount=0 before the patch and passes after.
10-12-2020
Fix request (13u) Requesting backport to 13u for parity with 11u. The patch doesn't apply cleanly since 13u doesn't have cgroups v2 support (JDK-8231111), reapplied manually. Tested with tier1 and container tests. RFR approval: http://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-December/004352.html
04-12-2020
8u review approval: https://mail.openjdk.java.net/pipermail/jdk8u-dev/2020-November/013039.html
26-11-2020
Fix Request (OpenJDK 11u): Please approve this backport for OpenJDK 11u. I see Oracle backported it too. The patch doesn't apply cleanly, since jdk head have the cgroups v2 refactoring. Patch has been reviewed by Aleksey Shipilev. Tested this on a system with swapaccount=0 which fails the test before and passes after. In general, this is a patch which improves resiliency so should be fairly low risk. No change for systems with swap accounting on; reports memory only - without swap - for systems with swap accounting off. Tier 1 tests pass too. webrev: https://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8250984/01/webrev/ RFR: http://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-October/003939.html
21-10-2020
Changeset: 01875677 Author: Harold Seigel <hseigel@openjdk.org> Date: 2020-09-25 17:16:38 +0000 URL: https://git.openjdk.java.net/jdk/commit/01875677
25-09-2020
What happens when running test jdk/internal/platform/docker/TestDockerMemoryMetrics.java is that method getStringValue(), in jdk/internal/platform/CgroupSubsystemController.java, gets an IO Exception: java.nio.file.NoSuchFileException: /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes, causing it to return null: public static String getStringValue(CgroupSubsystemController controller, String param) { if (controller == null) return null; try { return CgroupUtil.readStringValue(controller, param); } catch (IOException e) { return null; } } This method then returns -1, because strval is null: public static long getLongValue(CgroupSubsystemController controller, String param, Function<String, Long> conversion, long defaultRetval) { String strval = getStringValue(controller, param); if (strval == null) return defaultRetval; return conversion.apply(strval); } Control eventually passes back to getTotalSwapSpaceSize(), in src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java, which sees that limit is -1, so it returns the value for the system swap space, which, in this case, is 0. public long getTotalSwapSpaceSize() { if (containerMetrics != null) { long limit = containerMetrics.getMemoryAndSwapLimit(); // The memory limit metrics is not available if JVM runs on Linux host (not in a docker container) // or if a docker container was started without specifying a memory limit (without '--memory=' // Docker option). In latter case there is no limit on how much memory the container can use and // it can use as much memory as the host's OS allows. long memLimit = containerMetrics.getMemoryLimit(); if (limit >= 0 && memLimit >= 0) { return limit - memLimit; // might potentially be 0 for limit == memLimit } } return getTotalSwapSpaceSize0(); }
08-09-2020
Thanks. This seems to suggest that the host/system values are being reported as 0: OperatingSystemMXBean.getTotalSwapSpaceSize: 0 OperatingSystemMXBean.getFreeSwapSpaceSize: 0 That's different to what we have seen with JDK-8244500 in the past :(
01-09-2020
Metrics instance: non-null Metrics.getMemoryAndSwapLimit() == -1 Metrics.getMemoryLimit() == 104857600 Metrics.getMemoryAndSwapUsage() == -1 Metrics.getMemoryUsage() == 34107392
01-09-2020
There should be some traces about what the Java Metrics system thinks the limits are on such systems: From test/hotspot/jtreg/containers/docker/CheckOperatingSystemMXBean.java: System.out.println("Metrics instance: " + (metrics == null ? "null" : "non-null")); if (metrics != null) { System.out.println("Metrics.getMemoryAndSwapLimit() == " + metrics.getMemoryAndSwapLimit()); System.out.println("Metrics.getMemoryLimit() == " + metrics.getMemoryLimit()); System.out.println("Metrics.getMemoryAndSwapUsage() == " + metrics.getMemoryAndSwapUsage()); System.out.println("Metrics.getMemoryUsage() == " + metrics.getMemoryUsage()); } It would be good to know what it reports. This bug is very similar to JDK-8244500 and we've had several iterations on trying to getting it fixed.
01-09-2020
Test containers/docker/TestMemoryAwareness.java is failing with these results: OperatingSystemMXBean.getTotalSwapSpaceSize: 0 OperatingSystemMXBean.getFreeSwapSpaceSize: 0 stderr: [WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
28-08-2020
Runtime.availableProcessors: 4 [0.379s][trace][os,container] CgroupSubsystem::active_processor_count (cached): 4 OperatingSystemMXBean.getAvailableProcessors: 4 [0.388s][trace][os,container] Path to /cpu.cfs_quota_us is /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us [0.388s][trace][os,container] CPU Quota is: -1 [0.389s][trace][os,container] Path to /cpu.cfs_period_us is /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us [0.389s][trace][os,container] CPU Period is: 100000 [0.389s][trace][os,container] Path to /cpu.shares is /sys/fs/cgroup/cpu,cpuacct/cpu.shares OperatingSystemMXBean.getTotalMemorySize: 104857600 [0.389s][trace][os,container] CPU Shares is: 1024 [0.389s][trace][os,container] OSContainer::active_processor_count: 4 OperatingSystemMXBean.getTotalPhysicalMemorySize: 104857600 OperatingSystemMXBean.getFreeMemorySize: 69091328 [0.391s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes [0.391s][trace][os,container] Memory Limit is: 104857600 [0.391s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.391s][trace][os,container] Memory Usage is: 35762176 OperatingSystemMXBean.getFreePhysicalMemorySize: 69091328 [0.392s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.392s][trace][os,container] Memory Usage is: 35758080 OperatingSystemMXBean.getTotalSwapSpaceSize: 0 OperatingSystemMXBean.getFreeSwapSpaceSize: 0 [0.393s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.393s][trace][os,container] Memory Usage is: 35753984 [0.394s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.394s][trace][os,container] Memory Usage is: 35880960 [0.394s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.394s][trace][os,container] Memory Usage is: 35876864 [0.394s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.395s][trace][os,container] Memory Usage is: 35872768 [0.397s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.397s][trace][os,container] Memory Usage is: 35971072 [0.397s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.397s][trace][os,container] Memory Usage is: 35909632 [0.398s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.398s][trace][os,container] Memory Usage is: 35905536 OperatingSystemMXBean.getCpuLoad: 0.000000 [0.401s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.401s][trace][os,container] Memory Usage is: 35897344 OperatingSystemMXBean.getSystemCpuLoad: 0.666667 [0.402s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.402s][trace][os,container] Memory Usage is: 35893248 [0.403s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.403s][trace][os,container] Memory Usage is: 35889152 [0.404s][trace][os,container] Path to /memory.usage_in_bytes is /sys/fs/cgroup/memory/memory.usage_in_bytes [0.404s][trace][os,container] Memory Usage is: 35778560 ]; stderr: [WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. ] exitValue = 0 java.lang.RuntimeException: 'OperatingSystemMXBean\\.getFreeSwapSpaceSize: [1-9][0-9]+' missing from stdout/stderr at jdk.test.lib.process.OutputAnalyzer.shouldMatch(OutputAnalyzer.java:308) at TestMemoryAwareness.testOperatingSystemMXBeanAwareness(TestMemoryAwareness.java:159) at TestMemoryAwareness.main(TestMemoryAwareness.java:66) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298) at java.base/java.lang.Thread.run(Thread.java:832) JavaTest Message: Test threw exception: java.lang.RuntimeException JavaTest Message: shutting down test result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: 'OperatingSystemMXBean\.getFreeSwapSpaceSize: [1-9][0-9]+' missing from stdout/stderr
03-08-2020

Relates :	JDK-8244500 - jtreg test error in test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java
Relates :	JDK-8291570 - [TESTBUG] Part of JDK-8250984 absent from 11u
Relates :	JDK-8226575 - OperatingSystemMXBean should be made container aware
Relates :	JDK-8253797 - [cgroups v2] Account for the fact that swap accounting is disabled on some systems
Relates :	JDK-8300658 - memory_and_swap_limit() reporting wrong values on systems with swapaccount=0
Relates :	JDK-8264524 - jdk/internal/platform/docker/TestDockerMemoryMetrics.java fails due to swapping not working
Relates :	JDK-8227317 - [TESTBUG] jdk docker/TestDockerMemoryMetrics.java fails on systems w/o kernel memory accounting
Relates :	JDK-8257746 - Regression introduced with JDK-8250984 - memory might be null in some machines