JDK-8054108 : LargePages silently falling back to small pages with G1GC & 8GB+ heaps
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 7u60
  • Priority: P4
  • Status: Closed
  • Resolution: Duplicate
  • OS: linux
  • CPU: x86_64
  • Submitted: 2014-07-30
  • Updated: 2016-05-19
  • Resolved: 2016-04-25
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7 JDK 8 JDK 9
7-poolResolved 8-poolResolved 9Resolved
Related Reports
Duplicate :  
Description
tFULL PRODUCT VERSION :
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)

FULL OS VERSION :
2.6.18 x86_64

EXTRA RELEVANT SYSTEM CONFIGURATION :
Huge pages are manually configured and works correctly with JDK 6 and up to JDK 7u55 (inclusive).  Switching to 7u60 introduces the behavior of silently falling back

A DESCRIPTION OF THE PROBLEM :
Starting with 7u60 the combination of LargePages, G1GC, and a heap of 8GB or more causes the JVM to silently fall back to small pages.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

REGRESSION.  Last worked in version 7u60

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Super simple code to just let the JVM hang out so you can go to another window and check the hugepages utilization:

import javax.net.*;

class Hello {
    public static void main(String[] argv) throws Exception {
        System.out.println("Hello!");
        Thread.sleep(1000000);
    }
}

Works up to 1MB below the 8GB mark:

/apollo/env/BigBirdRequestRouterService/jdk/bin/java -Xms8191m -Xmx8191m -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+UseG1GC Hello
Hello!

(in another window)
cat /proc/meminfo |grep Huge
HugePages_Total:  8704
HugePages_Free:   8694
HugePages_Rsvd:   3622
Hugepagesize:     2048 kB

At 8GB (and higher) the hugepages just stop working and silently fall back to small pages:

/apollo/env/BigBirdRequestRouterService/jdk/bin/java -Xms8191m -Xmx8191m -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+UseG1GC Hello

cat /proc/meminfo |grep Huge
HugePages_Total:  8704
HugePages_Free:   8696
HugePages_Rsvd:      0
Hugepagesize:     2048 kB


However if I remove UseG1GC but leave the 8GB heap, it uses the huge pages:

java -Xms8g -Xmx8g -XX:+UseLargePages -XX:+AlwaysPreTouch Hello        
Hello!

cat /proc/meminfo |grep Huge
HugePages_Total:  8704
HugePages_Free:   4596
HugePages_Rsvd:    160
Hugepagesize:     2048 kB


EXPECTED VERSUS ACTUAL BEHAVIOR :
I expect the huge/large pages to be used even with G1GC and a heap of 8GB+
ERROR MESSAGES/STACK TRACES THAT OCCUR :
No errors, it silently falls back.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
See reproduction steps
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
I haven't been able to find one.


Comments
Fixed with JDK-8017629
25-04-2016

[~stefank] You are welcome to use this bug report if you want, or close it as a duplicate. Assigning it to you.
16-03-2016

A workaround to get large pages for the heap is to run with -XX:G1HeapRegionSize=2m. With this, the heap alignment will be the same as the large pages size, and we'll get past the alignment > os::large_page_size() check mentioned above.
15-03-2016

There's a similar bug to this that I'm currently working on and intend to fix: https://bugs.openjdk.java.net/browse/JDK-8017629 Maybe we should use this bug report and close the other as a duplicate, since this CR has more history.
15-03-2016

After disscussing this with Stefan we concluded falling back to small pages was expected behaviour. However, there should be some kind of warning that this happens and why.
07-11-2014

Forgot to mention that the check for if alignment is bigger than os::large_page_size is one of the more stringent heap alignment checks that Stefan K mentioned in the mail pasted above: if (!is_size_aligned(bytes, os::large_page_size()) || alignment > os::large_page_size())
23-10-2014

Can reproduce using -XX:+UseLargePages -XX:+UseSHM. When using less than 8GB of heap everything is fine: SHM: size: 8675917824, alignment: 2097152, os::large_page_size: 2097152 When setting it to 8GB or over the alignment increases to be bigger than os::large_page_size and large pages fail: SHM: size: 8673820672, alignment: 4194304, os::large_page_size: 2097152 Java HotSpot(TM) 64-Bit Server VM warning: Alignment error when setting up large pages. Looking at logic that determines alignment now.
23-10-2014

Stefan K commented on this in an email: It looks like they are using an old kernel that doesn't support hugetlbfs. > FULL OS VERSION : > 2.6.18 x86_64 $ man mmap: MAP_HUGETLB (since Linux 2.6.32) Allocate the mapping using "huge pages." See the Linux kernel source file Documentation/vm/hugetlbpage.txt for further information. So, they are probably using the older UseSHM system. My fix changed hugetlbfs but it also made Heap alignments more stringent with UseSHM (and UseTransparentHugePages). I can't explain what they are seeing, but it might be worth investigating more. For example, what happens if the run with 8GB + 1mb or 8gb + 2mb? Given this, I would say that ILW is: Impact: Medium, don't get large pages Likelihood: Low, happens on older kernels with specific heap and large pages settings Workaround: Medium-High, may be able to tune around it in some cases This gives P4 as priority. Still, it is a regression, so we should look at it for 7u80 to see if we can reproduce this on some machine with the UseSHM system. It may be that there is nothing we can do about it, or that the change will be to big for 7 at this stage
07-08-2014

Looks like this is manifesting due to the fix of https://bugs.openjdk.java.net/browse/JDK-8007074. Needs to be investigated if this is expected behavior or is really a bug.
02-08-2014