JDK-8222252 : Java ergonomics limits heap to 128GB with disabled compressed oops
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 11.0.2
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2019-03-28
  • Updated: 2019-06-18
  • Resolved: 2019-05-30
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 13 JDK 14
13 b24Fixed 14Fixed
Related Reports
CSR :  
JDK-8223957 - Improve the behavior of Max RAM settings and UseCompressedOops
Relates :  
JDK-6887571 - Increase default heap config sizes
Relates :  
JDK-8213175 - Java ergonomics limits heap to 32GB to allow compressed oops
Relates :  
JDK-8225104 - 32-bit build failures after JDK-8222252
Relates :  
JDK-8224764 - Create jtreg test for JDK-8222252
Sub Tasks
JDK-8224764 :  
Create jtreg test for JDK-8222252 - Resolved
JDK-8226901 :  
Release Note: Improve the Behavior of MaxRAM Settings and UseCompressedOops - Closed
Description
ADDITIONAL SYSTEM INFORMATION :
Linux, as provided by the openjdk docker image `openjdk:11-jre`

A DESCRIPTION OF THE PROBLEM :
When Java container ergonomic options are used to set MaxHeapSize, they are unable to scale beyond a container of 128GB of RAM.  For instance, when running `docker run -m200g -it --rm openjdk:11-jre java -XX:-UseCompressedOops -XX:MaxRAMPercentage=50 -version` Java only allocates a 64GB heap (50% of 128GB limit).

This appears to be due lines 1733-1735 of `src/hotspot/share/runtime/arguments.cpp`, where the effective physical memory is calculated as the minimum of the reported physical memory (in this case from the container limits) and the default value of MaxRAM (128GB):

```
  julong phys_mem =
    FLAG_IS_DEFAULT(MaxRAM) ? MIN2(os::physical_memory(), (julong)MaxRAM)
                            : (julong)MaxRAM;
```

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
docker run -m256g -it --rm openjdk:11-jre java -XX:-UseCompressedOops -XX:MaxRAMPercentage=100 -XshowSettings:vm -version

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
VM settings:
    Max. Heap Size (Estimated): 256.00G
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment (build 11.0.2+9-Debian-3bpo91)
OpenJDK 64-Bit Server VM (build 11.0.2+9-Debian-3bpo91, mixed mode)

ACTUAL -
VM settings:
    Max. Heap Size (Estimated): 128.00G
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment (build 11.0.2+9-Debian-3bpo91)
OpenJDK 64-Bit Server VM (build 11.0.2+9-Debian-3bpo91, mixed mode)


CUSTOMER SUBMITTED WORKAROUND :
When you know you're running in docker, you can set MaxRAM yourself using

-XX:MaxRAM=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)

FREQUENCY : always
Comments
Please see the attached file change-results-2.out to see the impact of my updated proposed change. This change only has impact if MaxRAMPercentage is specified. ./java -XX:MaxRAM=192g -XX:MaxRAMPercentage=75 -XX:+PrintFlagsFinal -version | grep MaxHeapSize size_t MaxHeapSize [ORIG] = 32178700288 {product} {ergonomic} size_t MaxHeapSize [NEW] = 154618822656 {product} {ergonomic} ./java -XX:MaxRAM=192g -XX:MaxRAMPercentage=75 -XX:+PrintFlagsFinal -version | grep UseCompressedOops bool UseCompressedOops [ORIG] = true {lp64_product} {ergonomic} bool UseCompressedOops [NEW] = false {lp64_product} {ergonomic} ./java -XX:MaxRAMPercentage=10 -XX:+PrintFlagsFinal -version | grep MaxHeapSize size_t MaxHeapSize [ORIG] = 13744734208 {product} {ergonomic} size_t MaxHeapSize [NEW] = 15183380480 {product} {ergonomic}
15-05-2019
Your suggestion is Linux specific and assumes that docker images have "cat" available. It also does not solve the problem reported in 8213175. I prefer a fix like this that solves both issues .... *** 1711,1723 **** // Use static initialization to get the default before parsing static const size_t DefaultHeapBaseMinAddress = HeapBaseMinAddress; void Arguments::set_heap_size() { ! julong phys_mem = ! FLAG_IS_DEFAULT(MaxRAM) ? MIN2(os::physical_memory(), (julong)MaxRAM) : (julong)MaxRAM; // Convert deprecated flags if (FLAG_IS_DEFAULT(MaxRAMPercentage) && !FLAG_IS_DEFAULT(MaxRAMFraction)) MaxRAMPercentage = 100.0 / MaxRAMFraction; --- 1711,1739 ---- // Use static initialization to get the default before parsing static const size_t DefaultHeapBaseMinAddress = HeapBaseMinAddress; void Arguments::set_heap_size() { ! julong phys_mem; ! bool use_os_mem_limit = false; ! ! // If the user specified one of these options, they ! // want memory limits calculated based on available ! // os physical memory, not our MaxRAM limit. ! if (!FLAG_IS_DEFAULT(MaxRAMPercentage) || ! !FLAG_IS_DEFAULT(MaxRAMFraction) || ! !FLAG_IS_DEFAULT(MinRAMPercentage) || ! !FLAG_IS_DEFAULT(MinRAMFraction) || ! !FLAG_IS_DEFAULT(InitialRAMPercentage) || ! !FLAG_IS_DEFAULT(InitialRAMFraction)) { ! ! phys_mem = os::physical_memory(); ! use_os_mem_limit = true; ! } else { ! phys_mem = FLAG_IS_DEFAULT(MaxRAM) ? MIN2(os::physical_memory(), (julong)MaxRAM) : (julong)MaxRAM; + } *** 1769,1780 **** --- 1785,1808 ---- if (HeapBaseMinAddress + MaxHeapSize < max_coop_heap) { // Heap should be above HeapBaseMinAddress to get zero based compressed oops // but it should be not less than default MaxHeapSize. max_coop_heap -= HeapBaseMinAddress; } + + // If user specified flags prioritizing os physical + // memory limits, then disable compressed oops if + // limits exceed max_coop_heap and UseCompressedOops + // was not specified on the command line. + if (reasonable_max > max_coop_heap) { + if (FLAG_IS_ERGO(UseCompressedOops) && use_os_mem_limit) { + FLAG_SET_ERGO(bool, UseCompressedOops, false); + FLAG_SET_ERGO(bool, UseCompressedClassPointers, false); + } else { reasonable_max = MIN2(reasonable_max, max_coop_heap); } + } + } reasonable_max = limit_by_allocatable_memory(reasonable_max);
10-05-2019
As workaround I meant in addition to setting MaxRAM, also have the user set MaxRAM appropriately, i.e. -XX:MaxRAM=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes) as a short-term workaround/fix. I agree with you that it is awkward to have to specify two options, and one of them being the result of a platform dependent script. Thanks for looking into this.
10-05-2019
Unfortunately, we could not close this issue with the work-around of specifying MaxRAM because this is not what the user is after. The folks using containers want to specify the amount of memory that the container can use and want to pass a generic option to Java to use a fixed % of that memory for the heap. If we don't make any changes to the implementation and pass -XX:MaxRAM=xxx along with -XX:MaxRAMPercentage=yyy, the heap size that will be selected will be yyy% of xxx instead of yyy% of os::physical_memory. I'll take a stab at producing a webrev and a table of inputs and results before and after. Then we can have a discussion.
08-05-2019
1) My suggestion is just that, which does need wider discussion - it would achieve the "expected" results at least. I would also be okay that if the user specified MaxRAM, in most cases it does not make a lot of sense without it (looking at the implementation). But the current behavior is a bit surprising, at least apparently for the reporter of this issue. However if we accepted this, it means we could close this issue as "works as expected" (as you suggested earlier), and the same for JDK-8213175. I think (without testing) if the reporter also specified MaxRAM like in the presented workaround for JDK-8213175 it would work too. 2) did some digging, I agree that the comment is wrong. However I think this is only some weird way to ensure that compressed oops are the preferred case if no other options are set (not sure!). I.e. that with no other options we limit the max heap size to < 32gb to get compressed oops. I still think this behavior is a good idea even if there is a huge amount of physical memory available for the obvious performance reasons. *) looking back at the other post, I am not sure about that changing the behavior in your previous suggestion at the end is a good idea. Ie: "This would allow someone to run this command in a 200GB container: java -XX:MaxRAM=1024G -XX:MaxRAMPercentage=50%." This would mean that the VM would silently override the user input (as far as I understand your code suggestion this would cap the phys_mem to 200G anyway, resulting in a 100g heap) - not really nice imho as I kind of prefer not doing that - I think we also allow an -Xmx larger than whatever phys_mem is in the calculation failing sometime later. Which would introduce an additional inconsistency in flag handling. I recommend asking around what others think and what you prefer. Or just ask them what they think the heap size should be for the various cases. :)
08-05-2019
I like your solution. The only issues I can think of are: 1. We are changing behavior since we can now exceed the safety belt default MaxRAM size by using MaxRAMPercentage. 2. I think set_use_compressed_oops will need to be updated since the assumption below is not valid. I don't believe it ever was. UseCompressedOops should be determined once all other heap size calculations are performed. If the user specified it (!FLAG_IS_DEFAULT), then this should factor into the heap size calulations by limiting it. " // MaxHeapSize is not set up properly at this point, but // the only value that can override MaxHeapSize if we are // to use UseCompressedOops is InitialHeapSize."
08-05-2019
One could make the MaxRAMPercentage setting to simply set MaxRAM, which then determines the heap size, and CompressedOops follows. I.e. MaxRAM sets the maximum physical limit (I assume that os::physical_memory() just returns the container limit); MaxRAMPercentage internally sets MaxRAM, and from that we calculate CompressedOops setting. If none of the MaxRAM* options are set, operate as before. This would solve both issues I think; i.e. on the one hand keep the existing behavior if you do not manually set MaxRAM* (with the safety belt), on the other hand if the user wants to use a specific amount or percentage of memory, have the VM adapt to it. E.g. if physical memory is 256g: java -XX:MaxRAM=192g Test -> max heap size is 192g, no compressed oops java -XX:MaxRAM=25.6g Test -> max heap size is 25.6g, compressed oops (ergonomically determined) java -XX:MaxRAMPercentage=75 Test -> max heap size is 192g, no compressed oops (This is JDK-8213175 as far as I understand; I do not think it is worth creating an extra fix just for that, that's why it's a "duplicate" to me) java -XX:MaxRAMPercentage=10 Test -> max heap size is 25.6g, compressed oops (ergonomically determined) java Test -> max heap is 32g-Epsilon, compressed oops ("safety belt case"; same as before, if the user does not specify otherwise, prefer compressed oops) java -XX:-UseCompressedOops Test -> max heap is 128g ("safety belt case"; using the default MaxRAM value; same as before) java -XX:+UseCompressedOops -XX:MaxRAM=128g Test -> error because impossible java -XX:+UseCompressedOops -XX:MaxRAMPercentage=75 Test -> same as above The cases with -XX:-UseCompressedOops are no issue I think. I did not play through all combinations, but if you think of MaxRAMPercentage as another way of specifying MaxRAM it seems to make sense (to me at least). Also, MaxRAMPercentage is defined as "Maximum percentage of real memory used for maximum heap size", i.e. the suggestion above corresponds better to the definition imho. This should be possible to implement. I think this behavior was as intended initially as you mentioned, but this behavior is nowhere documented, actually specified in a CSR, and verified in a test (which is part of the reason why we have these issues); all that has been the main complaint in the initial review thread apart from that it broke the "safety belt" cases ;) What do you think?
08-05-2019
Thomas, How did we go from "... it should not be lifted and works as designed (brought up by various people in internal discussion)" to, this is a duplicate of https://bugs.openjdk.java.net/browse/JDK-8213175? If it's the opinion of the GC team that MaxRAM should override, then I will close this bug. This issue has nothing to do with the other bug since the percentage calculation is being done correctly but is being capped by MaxRAM. Perhaps we could consider the following changes that would allow users to override MaxRAM but still provide setting the heap size as a percentage of the containers memory limit. phys_mem should never be set higher than the amount of available memory anyway, right? - julong phys_mem = - FLAG_IS_DEFAULT(MaxRAM) ? MIN2(os::physical_memory(), (julong)MaxRAM) - : (julong)MaxRAM; + julong phys_mem = MIN2(os::physical_memory(), (julong)MaxRAM); This would allow someone to run this command in a 200GB container: java -XX:MaxRam=1024G -XX:MaxRAMPercentage=50%.
08-05-2019
This looks to be a duplicate of JDK-8213175 - i.e. determination of the max physical memory does not respect the new *RAMPercentage options, which is already assigned to [~bobv] - assigning this one to him too.
24-04-2019
When using MaxRAMPercentage (or any other of the MaxRAM* flags) we might not want to use MaxRAM as limit. Since the current value of MaxRAM is a kind of safety belt to not use excessive amount of memory by default, it should not be lifted and works as designed (brought up by various people in internal discussion)
12-04-2019
Potential fix: set the pd_global value for MaxRAM to (size_t)-1 (or a similarly high value) for all configs.
11-04-2019
In JDK-6887571 the default value for "server" configurations has been updated to 128G.
11-04-2019
Moving to GC team. This doesn't seem container specific but due to the use of MaxRAM in calculating "physical memory" - if os::physical_memory() < MaxRAM then MaxRAM will be used, regardless of how os::physical_memory() was determined.
11-04-2019