We saw an issue where native memory usage out of the box on large machines with many cores had increased a lot between JDK 7 and JDK 8:
jdk 7 uses about 550 mb of memory
jdk 8 goes up to 1.5 gb of memory
The issue seemed to be that with tiered compilation and many compiler threads, we compiled a lot more methods in JDK 8 which lead to a lot bigger native memory footprint.
The native memory usage was reduced by running with:
-XX:CICompilerCount=2' -XX:-TieredCompilation -XX:CompileThreshold=80000 -XX:ReservedCodeCacheSize=48m
We never tried each of these flags separately in order to find out which gave the biggest impact.
We should look over if any of our default values gets too big on large machines with a lot of memory and CPUs.