Bug ID: JDK-8257032 Native memory leak in java.util.zip caused by mis-using the API

Type: Enhancement
Component: core-libs
Sub-Component: java.util.jar
Affected Version: 8u271,16

Priority: P4
Status: Open
Resolution: Unresolved
OS: linux
CPU: x86_64

Submitted: 2020-11-24
Updated: 2025-01-27

Other
tbdUnresolved

ADDITIONAL SYSTEM INFORMATION :
Java 1.8 271

A DESCRIPTION OF THE PROBLEM :
The API of InflaterInputStream and DeflaterOutputStream is misleading

Both of this stream classes have plain one-arg ctors that allocate Inflater/Deflater internally, and in that case closing the stream also invokes an "end" method on the Inflater/Deflater objects, which releases native memory immediately


However when trying to control buffer size, we have to call a ternary ctor and allocate Inflater/Deflater externally

e.g.

new InflaterInputStream(in, new Inflater(), 256000)

new DeflaterOutputStream(os, new Deflater(), 32000)

In this case, closing the stream does not invoke the "end" method of Inflater/Deflater, as it assumes no ownership over these objects. Rather the "end" is invoked by the "finalizer" when the objects go through GC. In this case our system went into what seems like native memory leak.

The Inflater/Deflater rely on non-heap native memory, so not terminating them explicitly, defers the release of associated native memory to Full GC event. 
This is the #1 reason of reported issues of native memory problems in JVM as reported on the internet.

Suggested remedy:
1. Add a 2-arg ctor to InflaterInputStream and DeflaterOutputStream that accepts an input stream and buffer size - this will prevent falling into the trap of not calling "end" on Deflater/Inflater
2. Document this potential native memory problem on the ternary ctor and in Inflater/Deflater classes

We just spent two weeks on production failures because of this issue which is very hard to diagnose.
Other reports by same cause:
https://medium.com/swlh/native-memory-the-silent-jvm-killer-595913cba8e7
https://www.elastic.co/blog/tracking-down-native-memory-leaks-in-elasticsearch

Also notice that starting with JDK 25 (i.e. since JDK-8225763), `Inflater` and `Deflater` now implement `AutoCloseable` which makes it easy to control their lifecycle in a try-with-resources block. The new `Inflater::close()` and `Deflater:close()` methods simply call `end()` and can be treated as synonyms for the corresponding `Inflater::end()` and `Deflater:end()` methods
27-01-2025
[~ysr], the issue with `close()` is that it behaves differently depending on how the stream was created. If the stream was created without a pre-existing Inflater/Deflater (i.e. the stream creates the Inflater/Deflater itself when constructed) then `close()` will also close (i.e. call `end()`) on the corresponding Inflater/Deflater. However, if the stream is created with a pre-existing Inflater/Deflater, then the stream doesn't take ownership of the Inflater, and doesn't close it (i.e. call `end()`) in its `close()` method. You can e.g. easily see this in the source code of `InfalterInputStream`: ``` boolean usesDefaultInflater = false; /** * Creates a new input stream with a default decompressor and buffer size. * @param in the input stream / public InflaterInputStream(InputStream in) { this(in, in != null ? new Inflater() : null); usesDefaultInflater = true; } ... /* * Closes this input stream and releases any system resources associated * with the stream. * @throws IOException if an I/O error has occurred */ public void close() throws IOException { if (!closed) { if (usesDefaultInflater) inf.end(); in.close(); closed = true; } } ``` I think the API documentation on `close()` is at least misleading, because it claims that it will "releases any system resources associated with the stream". This is wrong in the case of a user-supplied Inflater/Deflater and should be mentioned in the API documentation.
27-01-2025
Naive question, not being familiar with this code. Is there an issue with the `close()` of custom [Inflater\|Deflater][Input\|Output]Stream disposing of the associated native resources, but instead of having to rely on finalization (pre JDK 10) and cleaning (post JDK 10) to do that work? If the user code can call `clean()` at the time of `close()`, why can the JDK code not do that for custom versions as well? There's probably a subtle reason why, and it would be good to explain that clearly in the API note. Otherwise, it would look to the naive reader such as me to be a bug in the API that the user needs to work around in the case of custom streams above.
24-01-2025
> One of the requests in the bug description bug is for the API docs to highlight that "end" needs to called. There is an API note in the class descriptions on this, it was added in Java 10 as part of the work to more away from finalizers. I think the submitter has a fair point that the four [Inflater\|Deflater][Input\|Output]Stream classes could have done a better job pointing out when it's the user's responsibility to call Deflater::end / Inflater::end. When the user picks a different constructor to control the buffer size, it's all too easy to miss the fact that the Deflater/Inflater is not managed. Perhaps we should also add an API note to any stream constructor taking a Deflater or Inflater, something like this: > API Note: > It is the responsibility of the caller to manage the lifecycle of the supplied Deflater instance. The created output stream does not call Deflater.end() when closed. A code snippet showing correct usage could also be helpful.
18-11-2024
Finalization is very problematic and we have been moving the usages to cleaner for some time. I'm not aware of any cases where the Cleaner has made things "worse".
12-12-2020
Alan, my understanding is that rewrites to use Cleaner instead of finalize() will be more efficient in cases where the resource is properly closed. BUT it should not make much difference when the resource becomes unreachable without being closed. The GC still needs to discover the unreachable objects and some java thread later needs to invoke the cleanup method. Has the switch to Cleaner in JDK 10 made the problem worse in some way?
11-12-2020
Inflater, Deflater and ZipFile were changed from using finalizers to the Cleaner mechanism in JDK 10. It would be useful to get feedback from those running with applications that don't call "end". One of the requests in the bug description bug is for the API docs to highlight that "end" needs to called. There is an API note in the class descriptions on this, it was added in Java 10 as part of the work to more away from finalizers.
11-12-2020
There is no change for the API in question. It is applicable to 16. Moved to JDK for more evaluations.
11-12-2020