JDK-8044725 : Bug in zlib 1.2.5 prevents inflation of some gzipped files (zlib 1.2.8 port)
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util.jar
  • Affected Version: 8u5
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: linux_redhat_6.0
  • CPU: x86
  • Submitted: 2014-06-02
  • Updated: 2017-01-20
  • Resolved: 2014-06-04
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7 JDK 8 JDK 9
7u80Fixed 8u20Fixed 9 b17Fixed
Related Reports
Relates :  
Relates :  
Relates :  
Description
FULL PRODUCT VERSION :
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux XXX.nyc.deshaw.com 2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11 17:27:00 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

Microsoft Windows [Version 6.1.7601]


A DESCRIPTION OF THE PROBLEM :
In https://bugs.openjdk.java.net/browse/JDK-7110149, the version of zlib used by the JVM was changed from 1.2.3 to 1.2.5.  Unfortunately, this exposes the JVM to a bug in zlib 1.2.5 that prevents inflation of a certain valid files.  This bug is fixed in zlib 1.2.8 by the following commit: https://github.com/madler/zlib/commit/51370f365607fe14a6a7a1a27b3bd29d788f5e5b

When the bug occurs, you will see a stack trace like the following:

Exception in thread "main" java.util.zip.ZipException: invalid distances set
	at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
	at java.io.FilterInputStream.read(FilterInputStream.java:107)
	at InflaterTest.main(InflaterTest.java:34)

Note that it's rare for the deflation code to compress a file in a way that actually trips the bug.  In an in-house application, we see this issue in 8 out of 17,000 compressed files.  However, we bumped into this within a couple weeks of testing jdk8, so I expect that other users who generate a lot of large gzipped files will also experience this issue.

I should note that this problem isn't in the Java code, but in the zlib code itself.  In fact, you can get the "invalid distances set" error on the test.gz file by compiling the zlib 1.2.5 source and running the native application on the test.gz file to attempt to decompress it.  Decompression will succeed if you're using the zlib 1.2.3 or zlib 1.2.8 source code, but fail with "invalid distances set" if you're using zlib 1.2.5.

REGRESSION.  Last worked in version 7u60

ADDITIONAL REGRESSION INFORMATION: 
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Any attempt to inflate (via a GZIPInputStream or some other mechanism) a file that contains data formatted in a way that zlib 1.2.5 can't process will reproduce the problem.  I have uploaded such a file to GitHub ("test.gz" at https://github.com/katznm/JavaZlib125Problem).  To exhibit the problem, download and compile the InflaterTest.java file (also at https://github.com/katznm/JavaZlib125Problem) via "javac InflaterTest.java".  Then run the application on the test.gz file.  

The application wants a URL, so you can either download the test.gz file and run it like this:

java InflaterTest file://localhost/your_path/test.gz

Or you can run it directly against the file in the GitHub repository, assuming you don't run afoul of any GitHub download limitations:

java InflaterTest https://github.com/katznm/JavaZlib125Problem/blob/master/test.gz?raw=true



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
If you use jdk7 (which uses zlib 1.2.3, which doesn't have the bug), the program will run successfully to completion, and print out the number of uncompressed bytes in the file.  It will look something like this:

$ java InflaterTest file://localhost/your_path/test.gz
Total uncompressed bytes: 136169060



ACTUAL -
If you run with JDK8, which uses zlib 1.2.5, you will see the following exception:

$ java InflaterTest file://localhost/your_path/test.gz
Exception in thread "main" java.util.zip.ZipException: invalid distances set
	at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
	at java.io.FilterInputStream.read(FilterInputStream.java:107)
	at InflaterTest.main(InflaterTest.java:34)


ERROR MESSAGES/STACK TRACES THAT OCCUR :
This is the exception you will see, caused by a bug in zlib 1.2.5:

Exception in thread "main" java.util.zip.ZipException: invalid distances set
	at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
	at java.io.FilterInputStream.read(FilterInputStream.java:107)
	at InflaterTest.main(InflaterTest.java:34)


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.net.URL;
import java.util.zip.GZIPInputStream;

/**
 * Demonstrate bug in zlib 1.2.5 that prevents inflation 
 * of gzipped files in some cases.
 *
 * <p>Run via "java InflaterTest gzipped_url".  If you're
 * running on one of the unusual files that causes the problem,
 * then if you're using jdk7, the program will print the number
 * of uncompressed bytes in the file, and then exit.  On jdk8,
 * it will fail with an exception.
 */
public class InflaterTest
{
   /**
    * args: gzipped_filename
    */
    public static void main(String[] args) throws Exception
    {
        if (args.length != 1) {
            System.err.println("args: gzipped_url");
            System.exit(1);
        }

        URL url = new URL(args[0]);

        byte[] buffer = new byte[1024];
        long totalRead = 0;
        try (GZIPInputStream in = new GZIPInputStream(
                url.openStream()))
        {
            int nr;
            while ((nr = in.read(buffer)) != -1) {
                totalRead += nr;
            }
        }

        System.out.println("Total uncompressed bytes: " + totalRead);
    }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
If you copy the jre/lib/amd64/libzip.so file from the JDK7u60 distribution to the JDK8u5 distribution, it appears to fix the problem (as this effectively replaces zlib 1.2.5 with zlib 1.2.3).  However, this may be more luck than anything else.  I could imagine that future builds of JDK8 will make this impossible, as it's not clear that libzip.so is intended to be binary-compatible between JDK7 and JDK8.




Comments
target for 6u101
23-02-2015

A good reason to update zlib libraries in 8u and 9 perhaps.
03-06-2014