FULL PRODUCT VERSION :
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
FULL OS VERSION :
Linux andrew-VirtualBox 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
EXTRA RELEVANT SYSTEM CONFIGURATION :
Ubuntu 14.04 Guest running in VirtualBox on OSX 10.11.5 host.
No other changes to standard Ubuntu configuration.
A DESCRIPTION OF THE PROBLEM :
When running a web application for a long period of time, and that application makes a large number of long-runing requests (> 1s), it will begin to leak memory.
Looking at the application with JConsole, all the following pools are limited, and none are growing:
* Heap Space
* Meta Space
* Code Cache
* Compressed Class Space
As such, I'm not entirely sure where all the extra memory is going. Looking at the application with JCmd and NMT, I see that over time, the Class/Malloc area grows. That is, given the following partial output from jcmd <pid> VM.native_memory summary:
- Class (reserved=23075KB, committed=15651KB)
(mmap: reserved=22528KB, committed=15104KB)
The "classes" number remains static, the "mmap" reserved & committed numbers remain static, but the "malloc" number grows over time, and does not decrease.
I have also observed that, after setting MALLOC_ARENAS_MAX to 2, when running on a system with glibc >= 2.16, that the malloc arenas seem to fragment over time, and that, after a couple of days, the amount of native heap space reported by pmap grows. For the sample application I have provided below, I have seen the *native* (*not* Java!) heap space grow to over 200MB used. This seems... incorrect.
THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No
THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes
REGRESSION. Last worked in version 7u80
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Check out the sample project from GitHub onto a machine with JRE 1.8.0_101.
cd into nacroleptic
Run "mvn clean package"
Run java -jar target/narcoleptic.jar
cd into ../devourer
Run "mvn clean package"
Download Apache JMeter 3.0 from http://jmeter.apache.org/download_jmeter.cgi
Extract the archive and start with the "bin/jmeter.sh" script.
Open the file "load_test.jmx" with JMeter.
Start the JMeter tests with CTRL+R (CMD+R on OSX)
Leave for *at least* 30 minutes, then come back and observe the amount of memory used by the process. It will have increased beyond what would be expected. Leave for several hours, maybe a couple of days. Memory usage will continue to increase.
EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected behaviour: Memory usage of the application remains stable.
Actual behaviour: Memory usage of the application increases over time, without stopping. It may slow, and *appear* to stop, but given a large enough time span, it will *alwasy* continue to increase.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
N/A - application does not crash unless placed inside a cgroup (for example) with a hard memory limit.
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
Please find the source code at: https://github.com/ipsi/memory-testing
This is the smallest test case I have been able to find, and I've been looking quite hard.
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
Revert to Java 7
Disable HotSpot completely with -Xint
Set the MALLOC_ARENAS_MAX environment variable to 2, which slows the memory growth, but does not halt it.