Bug ID: JDK-8370572 Cgroups hierarchical memory limit is not honored after JDK-8322420

JDK-8370572 : Cgroups hierarchical memory limit is not honored after JDK-8322420

Type: Bug
Component: hotspot
Sub-Component: runtime
Affected Version: 21.0.9,25,26

Priority: P4
Status: Resolved
Resolution: Fixed

Submitted: 2025-10-24
Updated: 2025-12-15
Resolved: 2025-10-30

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 26
26 b23Fixed

Related Reports

Causes :	JDK-8322420 - [Linux] cgroup v2: Limits in parent nested control groups are not detected
Duplicate :	JDK-8372616 - 8370572: Cgroups hierarchical memory limit is not honored after JDK-8322420
Relates :	JDK-8370966 - Create regression test for the hierarchical memory limit fix in JDK-8370572

Description

JDK-8322420 removed the handling of `hierarchical_memory_limit`:
https://github.com/openjdk/jdk/commit/55a7cf14453b6cd1de91362927b2fa63cba400a1#diff-8910f554ed4a7bc465e01679328b3e9bd64ceaa6c85f00f0c575670e748ebba9L118-L131

It seems to be the reason for reports that ECS tasks are no longer honoring the memory limit:
 https://github.com/adoptium/adoptium-support/issues/1293
 https://github.com/corretto/corretto-21/issues/135

We initially found this in Corretto 21.0.9. I have deployed `public.ecr.aws/amazoncorretto/amazoncorretto:21` image to ECS, Fargate, 2GB task, and overridden Docker command to `java,-XX:InitialRAMPercentage=25,-XX:MaxRAMPercentage=25,-Xlog:os+container=trace,-Xlog:gc+init,-version`, and it showed me this:

```
[0.002s][trace][os,container] total physical memory: 4037939200
...
[0.002s][trace][os,container] Memory Limit is: 9223372036854771712
[0.002s][debug][os,container] container memory limit ignored: 9223372036854771712, using host value 4037939200
...
[0.013s][info ][gc,init ] Heap Initial Capacity: 964M
[0.013s][info ][gc,init ] Heap Max Capacity: 964M
...
openjdk version "21.0.9" 2025-10-21 LTS
OpenJDK Runtime Environment Corretto-21.0.9.10.1 (build 21.0.9+10-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.9.10.1 (build 21.0.9+10-LTS, mixed mode, sharing)
```

So regardless this is 2G task, we detect 4G as the memory limit, which gives 25% heap of 1G, which leads to OOM later.

In fact, this also reproduces with JDK mainline nightly, with a similar result:

```
| [0.001s][trace][os,container] OSContainer::init: Initializing Container Support
| [0.001s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
| [0.001s][debug][os,container] Detected cgroups hybrid or legacy hierarchy, using cgroups v1 controllers
...
| [0.002s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes
| [0.002s][trace][os,container] Memory Limit is: 9223372036854771712
| [0.002s][debug][os,container] container memory limit ignored: 9223372036854771712, upper bound is 4037939200
...
| [0.019s][info ][gc,init     ] Heap Initial Capacity: 964M
| [0.019s][info ][gc,init     ] Heap Max Capacity: 964M
...
| openjdk version "26-testing" 2026-03-17
| OpenJDK Runtime Environment (build 26-testing-builds.shipilev.net-openjdk-jdk-b5938-20251024-1037)
| OpenJDK 64-Bit Server VM (build 26-testing-builds.shipilev.net-openjdk-jdk-b5938-20251024-1037, mixed mode, sharing)
```

If you ask how ECS configures cgroups, it would say this:

```
/sys/fs/cgroup/memory/memory.use_hierarchy:1
/sys/fs/cgroup/memory/memory.stat:hierarchical_memory_limit 2147483648
...
/sys/fs/cgroup/memory/memory.limit_in_bytes:9223372036854771712
```

So there is _no_ `memory.limit_in_bytes` set, but `hierarchical_memory_limit` is still there. New JDK code leans heavily on `memory.limit_in_bytes` and therefore sees no limit. Older JDK code used to look at `hierarchical_memory_limit` and worked fine. 

My brief reading suggests that walking the hierarchy and looking for `memory.limit_in_bytes` is fine for cgroups V2, but for cgroups V1 we should still rely on `hierarchical_memory_limit`. I have not been able to see clear docs on this.

Comments

Fix Request (OpenJDK 25u): Please approve backporting this regression fix to OpenJDK 25. JDK-8322420 in JDK 21+ which caused the regression on cgroups v1 systems. In particular ECS Fargate systems[1] on cgroups v1 are affected (the only workaround is to apply a hard limit). This restores the behavour to prior JDK-8322420 in terms of physical memory detection. The JDK head fix doesn't apply cleanly but was fairly trivially resolved. Reviewed by Paul Hohensee. The fix is Linux-only and cgroups v1 only and only comes into play if no other limit is being detected. Thus, it should be a low risk fix. Tested with the regression test in JDK-8370966 which applies clean and is waiting for approval as well. [1] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html
12-12-2025
The proposed JDK 25u backport should be fairly clean for jdk21u-dev. PTAL, [~shade]. Once JDK 25u backport is in I'll proceed with a JDK 21u version.
11-12-2025
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk25u-dev/pull/76 Date: 2025-12-11 11:38:27 +0000
11-12-2025
I have to disagree. It's not a goal to keep JDK trees in sync. jlong/julong changes aren't suitable for backport. Maybe they are at some point, but not now.
28-11-2025
My original plan was to pull it into 25u and 21u ASAP. But it proves to be hairy to backport cleanly, as there were several prior changes in related code, which risks destabilizing this whole thing. So, it will unfortunately come slowly. We took the calculated risk with Corretto 21.0.9.11.1 and 25.0.1.9.1, where we pulled related patches along with this fix. If you are facing this issue, consider testing those releases and see if magic smoke is coming out of them. :) For JDK 25u, the clean backport stack looks like this: Clean JDK-8370966: Create regression test for the hierarchical memory limit fix in JDK-8370572 Clean JDK-8370572: Cgroups hierarchical memory limit is not honored after JDK-8322420 Clean JDK-8292984: Refactor internal container-related interfaces for clarity Clean JDK-8356868: Not all cgroup parameters are made available Clean JDK-8367485: os::physical_memory is broken in 32-bit JVMs when running on 64-bit OSes Clean JDK-8357086: os::xxx functions returning memory size should return size_t Clean JDK-8357570: [macOS] os::Bsd::available_memory() might return too low values Clean JDK-8364248: Separate commit and reservation limit detection
28-11-2025
We attempted a surgical backport in Corretto and failed; we had to bring in more things along the way. I am thinking we should at least attempt to backport container-code-related changes. This would also make sure we do not diverge the JDK versions (and thus introduce release-specific bugs). I am currently working through 25u stack, looking how bad that really is. 21u stack would likely be unclean no matter how hard we try, so we would have to make some concessions there...
28-11-2025
If backporting, then it'll have to be done surgically. I don't see a clean path for it. The fix is small enough to do stand-alone. JDK-8357086 and JDK-8367485 should definitely NOT be backported.
28-11-2025
Changeset: c49a94bf Branch: master Author: Aleksey Shipilev <shade@openjdk.org> Date: 2025-10-30 19:09:34 +0000 URL: https://git.openjdk.org/jdk/commit/c49a94bf89876c4d6c777a9452618afa564c5c23
30-10-2025
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk/pull/28006 Date: 2025-10-27 17:40:45 +0000
27-10-2025
I attached JDK-8370572-poc-1.patch that restores hunks of JDK-8322420 related to `hierarchical_memory_limit`. It seems to solve the problem with local Docker reproducer and on ECS as well.
24-10-2025
That's what I am saying: it looks like container got itself a child of the outside parent, and that child is now _root_ from the perspective of container. There is no "parent" from within the container anymore, only a single "child" one? So the full hierarchy is not visible to the container, but kernel still computes limits for us, because it _can_ still see it. % find /sys/fs/cgroup 2>&1 \| grep memory.stat ... /sys/fs/cgroup/memory/memory/parent/memory.stat /sys/fs/cgroup/memory/memory/memory.stat % docker run --cgroup-parent=/parent --rm -it shipilev/openjdk:latest find /sys/fs/cgroup \| grep memory.stat /sys/fs/cgroup/memory/memory.stat
24-10-2025
Yeah. This is interesting: [0.001s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes it should be: /sys/fs/cgroup/memory/parent/memory.limit_in_bytes
24-10-2025
Yes. So I am thinking we should trust `hierarchical_memory_limit`, if it is available; because that is what kernel computes for us, regardless of the actually visible hierarchy from inside of container? But maybe there is another solution I am not seeing.
24-10-2025
The container detection is a heuristic and it's not guaranteed to be always right. Question is how to best deal with this.
24-10-2025
Already did, same thing. My shipilev/openjdk:21 (jdk21u-dev nightly) fails on both ECS and locally in the reproducer above. Note that my reproducer above is failing even with current mainline, so it does not look like a missing backport. And the cgroup config that container sees seems to be similar to what container sees when running on ECS. That I think is the culprit...
24-10-2025
Please test 21.0.10 (21u-dev) as it also includes JDK-8343191.
24-10-2025
> Is the a reason why ECS sets up the cgroup subsystem the way it does - using the hierarchical limit rather than the real one? We are still following up on that. But I see the problem even with plain Docker, see above. Seems to be a generic V1 / Docker compatibility problem to me? E.g. it looks like if part of cgroup hierarchy is not visible (that's what cgroup-parent does, AFAIU), then our hierarchy walk does not see whatever limit we set "outside" of container. But the kernel reports it right via hierarchical_memory_limit...
24-10-2025
Found a local reproducer without ECS, just with plain Docker: % sudo mkdir /sys/fs/cgroup/memory/parent % echo 1073741824 \| sudo tee /sys/fs/cgroup/memory/parent/memory.limit_in_bytes % docker run --cgroup-parent=/parent --rm -it shipilev/openjdk:latest grep -rH . /sys/fs/cgroup/ \| grep -E \(memory.limit_in_bytes\\|hierarch\) /sys/fs/cgroup/memory/memory.use_hierarchy:1 /sys/fs/cgroup/memory/memory.stat:hierarchical_memory_limit 1073741824 /sys/fs/cgroup/memory/memory.limit_in_bytes:9223372036854771712 % docker run --cgroup-parent=/parent shipilev/openjdk:latest java -Xlog:os+container=trace -XX:InitialRAMPercentage=25 -XX:MaxRAMPercentage=25 -Xlog:gc+init -version [0.001s][trace][os,container] OSContainer::init: Initializing Container Support [0.001s][debug][os,container] Detected optional pids controller entry in /proc/cgroups [0.001s][debug][os,container] Detected cgroups hybrid or legacy hierarchy, using cgroups v1 controllers [0.001s][debug][os,container] OSContainer::init: is_containerized() = true because all controllers are mounted read-only (container case) ... [0.001s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes [0.001s][trace][os,container] Memory Limit is: 9223372036854771712 [0.001s][debug][os,container] container memory limit ignored: 9223372036854771712, upper bound is 264567476224 ... [0.166s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes [0.166s][trace][os,container] Memory Limit is: 9223372036854771712 [0.166s][debug][os,container] container memory limit ignored: 9223372036854771712, upper bound is 264567476224 [0.166s][info ][gc,init ] Memory: 246G ... [0.166s][info ][gc,init ] Heap Initial Capacity: 63104M [0.166s][info ][gc,init ] Heap Max Capacity: 63104M ... openjdk version "26-testing" 2026-03-17 OpenJDK Runtime Environment (build 26-testing-builds.shipilev.net-openjdk-jdk-b5938-20251024-1037) OpenJDK 64-Bit Server VM (build 26-testing-builds.shipilev.net-openjdk-jdk-b5938-20251024-1037, mixed mode, sharing)
24-10-2025
> I am thinking we should reinstate `hierarchical_memory_limit` check? It would make the container subsystem code even more complicated than it already is. So ECS with cgroup v1 relies on a systemd-related fix which was ineffective on cgroup v2: JDK-8217338. Interesting. Is the a reason why ECS sets up the cgroup subsystem the way it does - using the hierarchical limit rather than the real one? Are there ECS systems with cgroup v2? If so how does the setup look like there for the same system? We know setting the hard memory limit fixes the issue.
24-10-2025
[~sgehwolf], any thoughts? I am thinking we should reinstate `hierarchical_memory_limit` check?
24-10-2025