United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-5092263 GZIPInputStream spuriously reports "Corrupt GZIP trailer" for sizes > 2GB
JDK-5092263 : GZIPInputStream spuriously reports "Corrupt GZIP trailer" for sizes > 2GB

Details
Type:
Bug
Submit Date:
2004-08-26
Status:
Resolved
Updated Date:
2005-04-01
Project Name:
JDK
Resolved Date:
2005-04-01
Component:
core-libs
OS:
solaris_10,windows_xp
Sub-Component:
java.util.jar
CPU:
x86,sparc
Priority:
P3
Resolution:
Fixed
Affected Versions:
5.0,6
Fixed Versions:

Related Reports
Backport:
Backport:
Duplicate:
Relates:
Relates:

Sub Tasks

Description
When inflating a file whose inflated size is 2GB or more, GZIPInputStream
incorrectly throws IOException.

The test program below compresses the specified number of double-words, then
decompresses and verifies.   Examples:

$ java -server GzipTest file-64KB.gz 8192
File file-64KB.gz written.

$ java -server GzipTest file-2GB.gz 268435456
File file-2GB.gz written.
Exception in thread "main" java.lang.Error: Error reading
	at GzipTest.verify(GzipTest.java:82)
	at GzipTest.main(GzipTest.java:26)
Caused by: java.io.IOException: Corrupt GZIP trailer
	at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:174)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:89)
	at GzipTest.verify(GzipTest.java:67)
	... 1 more


$ cat GzipTest.java
import java.io.*;
import java.nio.ByteBuffer;
import java.util.zip.*;

public class GzipTest {

    private static final byte[] PI;
    static {
        ByteBuffer bb = ByteBuffer.allocate(8);
        bb.putDouble(Math.PI);
        PI = bb.array();
    }

    private final File file;
    private final int count;

    public static void main(String[] args) {
        if (args.length != 2) {
            System.err.println("Usage: GzipTest <filename.gz> <count>");
            System.exit(1);
        } 
        File outputFile = new File(args[0]);
        GzipTest test = new GzipTest(outputFile, Integer.parseInt(args[1]));
        test.write();
        System.out.println("File " + args[0] + " written.");
        if (test.verify()) {
            System.err.println("Verification FAILED!");
        }
    }

    public GzipTest(File file, int count) {
        this.file = file;
        this.count = count;
    }

    /** Writes count copies of PI to a compressed file. */
    public void write() {
        GZIPOutputStream gzipper;
        try {
            gzipper = new GZIPOutputStream(new FileOutputStream(file));
        } catch (IOException error) {
            throw new IllegalArgumentException("Error opening " + file);
        }
        try {
            for (int i = 0; i < count; i++) {
                gzipper.write(PI, 0, 8);
            }
            gzipper.close();
        } catch (IOException e) {
            throw new Error("Error writing", e);
        }
    }

    /** Returns true on error. */
    public boolean verify() {
        GZIPInputStream gunzipper;
        try {
            gunzipper = new GZIPInputStream(new FileInputStream(file));
        } catch (IOException error) {
            throw new IllegalArgumentException("Error opening " + file);
        }
        byte[] input = new byte[8];
        try {
            for (int i = 0; true; i++) {
                int pos = 0;
                do {
                    int read = gunzipper.read(input, pos, input.length - pos);
                    if (read < 0) {
                        gunzipper.close();
                        return i != count;
                    }
                    pos += read;
                } while (input.length - pos > 0);
                for (int j = 0; j < input.length; j++) {
                    if (input[j] != PI[j]) {
                        gunzipper.close();
                        return true;
                    }
                } 
            }     
        } catch (IOException e) {
            throw new Error("Error reading", e);
        }
    }
}

                                    

Comments
EVALUATION

Here is a somewhat more minimal test case:

----------------------------------------------------------------------------
import java.io.*;
import java.nio.ByteBuffer;
import java.util.zip.*;

public class GzipTest {
    static void readFully(InputStream s, byte[] input) throws Exception {
	int pos = 0;
	int n;
	while ((n = s.read(input, pos, input.length-pos)) > 0)
	    pos += n;
	if (pos != input.length)
	    throw new Exception("Unexpected EOF");
    }

    public static void main(String[] args) throws Exception {
        if (args.length != 2)
	    throw new Exception("Usage: GzipTest FILENAME COUNT");

	File file = new File(args[0]);
	int count = Integer.parseInt(args[1]);

	ByteBuffer bb = ByteBuffer.allocate(8);
        bb.putDouble(Math.PI);
        byte[] PI = bb.array();

	// Write count copies of PI to a compressed file
	GZIPOutputStream out = new GZIPOutputStream(new FileOutputStream(file));
	for (int i = 0; i < count; i++)
	    out.write(PI, 0, 8);
	out.close();

	// Verify file
	GZIPInputStream in = new GZIPInputStream(new FileInputStream(file));
	byte[] maybePI = new byte[8];
	for (int i = 0; i < count; i++) {
	    readFully(in, maybePI);
	    for (int j = 0; j < PI.length; j++)
		if (maybePI[j] != PI[j])
		    throw new Exception("data corruption");
	}
	if (in.read(maybePI, 0, 1) > 0)
	    throw new Exception("Unexpected NON-End of File");
    }
}
----------------------------------------------------------------------------
(martin@suttles) ~/src/toy/5092263 $ jver 1.5 java -server GzipTest foo 268435456
Exception in thread "main" java.io.IOException: Corrupt GZIP trailer
	at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:175)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:89)
	at GzipTest.main(GzipTest.java:41)

The Gzip file format uses 4 byte fields for byte counts,
so those fields are only accurate modulo 2**32.

gzip 1.2.4 does not support files larger than 4GB.  You need
the (currently beta) gzip 1.3 for that.

###@###.### 2004-08-29

It's interesting to note that gzip had essentially the same bug
as the jdk, both for many years.
###@###.### 2005-2-15 19:21:30 GMT
                                     
2004-08-29
CONVERTED DATA

BugTraq+ Release Management Values

COMMIT TO FIX:
mustang


                                     
2004-09-08
SUGGESTED FIX

In method readTrailer(), the integer returned by getTotalOut() is being
sign-extended to a long.   It ought to be zero-extended.

In particular, change:

    if (readUInt(in) != v || readUInt(in) != inf.getTotalOut()) {

to:

    if (readUInt(in) != v
              || readUInt(in) != (inf.getTotalOut() & 0xFFFFFFFFL)) {
    
    
----------------------------------------------------------------------
--- mustang/src/share/classes/java/util/zip/GZIPInputStream.java	2004-08-27 15:55:08.409027000 -0700
+++ 2g/src/share/classes/java/util/zip/GZIPInputStream.java	2005-02-14 23:40:13.422812000 -0800
@@ -170,10 +170,11 @@
 	    in = new SequenceInputStream(
 			new ByteArrayInputStream(buf, len - n, n), in);
 	}
-	long v = crc.getValue();
-	if (readUInt(in) != v || readUInt(in) != inf.getTotalOut()) {
+	// Uses left-to-right evaluation order
+	if ((readUInt(in) != crc.getValue()) ||
+	    // rfc1952; ISIZE is the input size modulo 2^32
+	    (readUInt(in) != (inf.getBytesWritten() & 0xffffffffL)))
 	    throw new IOException("Corrupt GZIP trailer");
-	}
     }
 
     /*

###@###.### 2005-2-15 07:45:07 GMT
                                     
2005-02-15



Hardware and Software, Engineered to Work Together