JDK-5092263 : GZIPInputStream spuriously reports "Corrupt GZIP trailer" for sizes > 2GB
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util.jar
  • Affected Version: 5.0,6
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: solaris_10,windows_xp
  • CPU: x86,sparc
  • Submitted: 2004-08-26
  • Updated: 2005-04-01
  • Resolved: 2005-04-01
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other JDK 6
1.4.2_12Fixed 6 b31Fixed
Related Reports
Duplicate :  
Relates :  
Relates :  
Description
When inflating a file whose inflated size is 2GB or more, GZIPInputStream
incorrectly throws IOException.

The test program below compresses the specified number of double-words, then
decompresses and verifies.   Examples:

$ java -server GzipTest file-64KB.gz 8192
File file-64KB.gz written.

$ java -server GzipTest file-2GB.gz 268435456
File file-2GB.gz written.
Exception in thread "main" java.lang.Error: Error reading
	at GzipTest.verify(GzipTest.java:82)
	at GzipTest.main(GzipTest.java:26)
Caused by: java.io.IOException: Corrupt GZIP trailer
	at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:174)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:89)
	at GzipTest.verify(GzipTest.java:67)
	... 1 more


$ cat GzipTest.java
import java.io.*;
import java.nio.ByteBuffer;
import java.util.zip.*;

public class GzipTest {

    private static final byte[] PI;
    static {
        ByteBuffer bb = ByteBuffer.allocate(8);
        bb.putDouble(Math.PI);
        PI = bb.array();
    }

    private final File file;
    private final int count;

    public static void main(String[] args) {
        if (args.length != 2) {
            System.err.println("Usage: GzipTest <filename.gz> <count>");
            System.exit(1);
        } 
        File outputFile = new File(args[0]);
        GzipTest test = new GzipTest(outputFile, Integer.parseInt(args[1]));
        test.write();
        System.out.println("File " + args[0] + " written.");
        if (test.verify()) {
            System.err.println("Verification FAILED!");
        }
    }

    public GzipTest(File file, int count) {
        this.file = file;
        this.count = count;
    }

    /** Writes count copies of PI to a compressed file. */
    public void write() {
        GZIPOutputStream gzipper;
        try {
            gzipper = new GZIPOutputStream(new FileOutputStream(file));
        } catch (IOException error) {
            throw new IllegalArgumentException("Error opening " + file);
        }
        try {
            for (int i = 0; i < count; i++) {
                gzipper.write(PI, 0, 8);
            }
            gzipper.close();
        } catch (IOException e) {
            throw new Error("Error writing", e);
        }
    }

    /** Returns true on error. */
    public boolean verify() {
        GZIPInputStream gunzipper;
        try {
            gunzipper = new GZIPInputStream(new FileInputStream(file));
        } catch (IOException error) {
            throw new IllegalArgumentException("Error opening " + file);
        }
        byte[] input = new byte[8];
        try {
            for (int i = 0; true; i++) {
                int pos = 0;
                do {
                    int read = gunzipper.read(input, pos, input.length - pos);
                    if (read < 0) {
                        gunzipper.close();
                        return i != count;
                    }
                    pos += read;
                } while (input.length - pos > 0);
                for (int j = 0; j < input.length; j++) {
                    if (input[j] != PI[j]) {
                        gunzipper.close();
                        return true;
                    }
                } 
            }     
        } catch (IOException e) {
            throw new Error("Error reading", e);
        }
    }
}

Comments
SUGGESTED FIX In method readTrailer(), the integer returned by getTotalOut() is being sign-extended to a long. It ought to be zero-extended. In particular, change: if (readUInt(in) != v || readUInt(in) != inf.getTotalOut()) { to: if (readUInt(in) != v || readUInt(in) != (inf.getTotalOut() & 0xFFFFFFFFL)) { ---------------------------------------------------------------------- --- mustang/src/share/classes/java/util/zip/GZIPInputStream.java 2004-08-27 15:55:08.409027000 -0700 +++ 2g/src/share/classes/java/util/zip/GZIPInputStream.java 2005-02-14 23:40:13.422812000 -0800 @@ -170,10 +170,11 @@ in = new SequenceInputStream( new ByteArrayInputStream(buf, len - n, n), in); } - long v = crc.getValue(); - if (readUInt(in) != v || readUInt(in) != inf.getTotalOut()) { + // Uses left-to-right evaluation order + if ((readUInt(in) != crc.getValue()) || + // rfc1952; ISIZE is the input size modulo 2^32 + (readUInt(in) != (inf.getBytesWritten() & 0xffffffffL))) throw new IOException("Corrupt GZIP trailer"); - } } /* ###@###.### 2005-2-15 07:45:07 GMT
15-02-2005

CONVERTED DATA BugTraq+ Release Management Values COMMIT TO FIX: mustang
08-09-2004

EVALUATION Here is a somewhat more minimal test case: ---------------------------------------------------------------------------- import java.io.*; import java.nio.ByteBuffer; import java.util.zip.*; public class GzipTest { static void readFully(InputStream s, byte[] input) throws Exception { int pos = 0; int n; while ((n = s.read(input, pos, input.length-pos)) > 0) pos += n; if (pos != input.length) throw new Exception("Unexpected EOF"); } public static void main(String[] args) throws Exception { if (args.length != 2) throw new Exception("Usage: GzipTest FILENAME COUNT"); File file = new File(args[0]); int count = Integer.parseInt(args[1]); ByteBuffer bb = ByteBuffer.allocate(8); bb.putDouble(Math.PI); byte[] PI = bb.array(); // Write count copies of PI to a compressed file GZIPOutputStream out = new GZIPOutputStream(new FileOutputStream(file)); for (int i = 0; i < count; i++) out.write(PI, 0, 8); out.close(); // Verify file GZIPInputStream in = new GZIPInputStream(new FileInputStream(file)); byte[] maybePI = new byte[8]; for (int i = 0; i < count; i++) { readFully(in, maybePI); for (int j = 0; j < PI.length; j++) if (maybePI[j] != PI[j]) throw new Exception("data corruption"); } if (in.read(maybePI, 0, 1) > 0) throw new Exception("Unexpected NON-End of File"); } } ---------------------------------------------------------------------------- (martin@suttles) ~/src/toy/5092263 $ jver 1.5 java -server GzipTest foo 268435456 Exception in thread "main" java.io.IOException: Corrupt GZIP trailer at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:175) at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:89) at GzipTest.main(GzipTest.java:41) The Gzip file format uses 4 byte fields for byte counts, so those fields are only accurate modulo 2**32. gzip 1.2.4 does not support files larger than 4GB. You need the (currently beta) gzip 1.3 for that. ###@###.### 2004-08-29 It's interesting to note that gzip had essentially the same bug as the jdk, both for many years. ###@###.### 2005-2-15 19:21:30 GMT
29-08-2004