JDK-6728376 : Wrong error handling in Java_java_util_zip_Deflater_deflateBytes leads to size 0 if compress fails
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util.jar
  • Affected Version: 6u7
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: solaris_10
  • CPU: generic
  • Submitted: 2008-07-22
  • Updated: 2011-07-27
  • Resolved: 2009-06-22
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other JDK 6 JDK 7
5.0u33Fixed 6u10Fixed 7 b62Fixed
Here is a small testcase that uses java.util.zip.Deflater

import java.util.Random;
import java.util.zip.Deflater; 

public class TestDeflater {

      public static void main(String[] args) {

            int max_byte_length = 200000000;
            byte[] inputData = new byte[max_byte_length];
            Random random = new Random();

            System.out.println("dataSize:" + inputData.length);

            Deflater compressor = new Deflater(Deflater.BEST_COMPRESSION);

            byte[] tempBuf = new byte[inputData.length];
            int compressedLength = compressor.deflate(tempBuf);

            System.out.println("compressedSize:" + compressedLength);


This is how it works normally:

$ java TestDeflater     

However, if either virtual memory (swap space) is scarce or the Java
process is about to hit the (4GB on Solaris32 / 2GB on Windows/Linux) 
process size limit, then we will see this:

$ ulimit -d 100000  //simulating low memory condition
$ java TestDeflater
compressedSize:0    <<<<< 

The expected behaviour should be to throw an OOME instead.

SUGGESTED FIX In Inflater.c and Deflater.c, change this code pattern: in_buf = (jbyte *) malloc(this_len); if (in_buf == 0) { return 0; } to: in_buf = (jbyte *) malloc(this_len); if (in_buf == 0) { JNU_ThrowOutOfMemoryError(env, 0); } Note that in some cases, previously-allocated memory should be freed before the OOME is thrown.

EVALUATION There are a few places in both Deflater.c and Inflater.c which have the problem noted in the comments. There are also places in those files from which an OOME is thrown via JNU_ThrowOutOfMemoryError(env, 0);