JDK-8054841 : (process) ProcessBuilder leaks native memory
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.lang
  • Affected Version: 7u67
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2014-08-08
  • Updated: 2016-06-13
  • Resolved: 2014-08-11
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7 JDK 8 JDK 9
7u72Fixed 8u40Fixed 9 b28Fixed
Related Reports
Relates :  
Description
FULL PRODUCT VERSION :
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
-------------------------------------------------------------------------------------------------
java version "1.7.0_65"
OpenJDK Runtime Environment (rhel-2.5.1.2.el7_0-x86_64 u65-b17)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux centos7-1 3.10.0-123.4.2.el7.x86_64 #1 SMP Mon Jun 30 16:09:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Have also repro'd on multiple versions of CentOS6, 64-bit

A DESCRIPTION OF THE PROBLEM :
A very detailed description, along with a simple reproducer and a graph of memory usage in various configurations, can be found here:

https://github.com/cprice404/jvm-processbuilder-leak/blob/master/README.md

Here is a slightly less detailed description:

We have noticed that in recent versions of the Oracle JDK and OpenJDK on 64-bit linux platforms, the ProcessBuilder class appears to leak some native memory whenever an external process is executed.

This bug does not appear to be present in OpenJDK 7u55, but does appear to be present in OpenJDK 7u65.
For the Oracle JDK, it does not appear to be present in 7u45, but does appear to be present in 7u55.

The problem can be reproduced by simply creating a loop that, for each iteration, uses ProcessBuilder to spawn a short-lived external process (we've reproduced with simple "echo" statements, or with calls to commands like "hostname").  Then, monitor the resident memory usage of the process reported by linux as the loop continues to run.

We have also tried variants where we take extra care to ensure that we consume and close the InputStreams for the external process's stdout/stderr, and variants where we circument ProcessBuilder and call into the lower-level "UNIXProcess" class directly.  In all cases, there appears to be no leak in older versions of the JDK but a very notable leak in the later versions.

The leak does not appear to be present on Mac OSX JDKs.


REGRESSION.  Last worked in version 7u65

ADDITIONAL REGRESSION INFORMATION: 
For OpenJDK:

java version "1.7.0_65"
OpenJDK Runtime Environment (rhel-2.5.1.2.el7_0-x86_64 u65-b17)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
-----------------------------------------------------------------------------------------------
For OracleJDK:

java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The easiest way is to simply clone the reproducer code that I've set up in this github repo:

https://github.com/cprice404/jvm-processbuilder-leak

and then run:

    mvn compile
    mvn exec:exec

Alternately you can just write some code that uses  ProcessBuilder in a loop, e.g.:

     ProcessBuilder pb = new ProcessBuilder("hostname");
     Process p = pb.start();
     p.waitFor();
     System.out.println(IOUtils.toString(p.getInputStream()));

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Spawning external processes in a loop, I would expect that resident memory usage would grow for a while until the Java max heap size (plus some small amount of additional memory for permgen and other memory) had been reached, and then resident memory usage would flatline and stay consistent for as long as the program was allowed to run.  This is precisely the behavior that I see with OpenJDK 7u55 or Oracle JDK 7u45.
ACTUAL -
Run the code above in a long-running loop, or running the maven example I've provided, on a 64-bit CentOS 6 or 7 system.  Using "top" or "ps -o rss" (or the "watchpidmem.sh" script included in my reproducer example), watch the resident memory usage of the process grow infinitely over time.  

In my reproducer the min and max heap size are set to 32MB, but on my systems that have 1GB RAM the process consistently grows to as much as 800MB and is eventually OOM-killed by linux.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
No Java error messags.  No relevant error messages at all other than a message in the syslog when linux eventually OOM-kills the process.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
while (true) {
    ProcessBuilder pb = new ProcessBuilder("hostname");
    Process p = pb.start();
    p.waitFor();    
}
---------- END SOURCE ----------


Comments
From the code analysis it can be seen that in src/solaris/native/java/lang/UNIXProcess_md.c: Java_java_lang_UNIXProcess_forkAndExec() bytes of a byte array are obtained by phelperpath = getBytes(env, helperpath), but are never released by releaseBytes().
11-08-2014