JDK-4418997 : Files between 2 Gb and 4Gb (excluded) are not accepted in a zip file.
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util.jar
  • Affected Version: 1.1.6,1.3.0
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • OS: windows_nt
  • CPU: x86
  • Submitted: 2001-02-26
  • Updated: 2017-05-16
  • Resolved: 2003-09-12
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
5.0 tigerFixed
Related Reports
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Description

Name: boT120536			Date: 02/26/2001


Development, JDK: "java version JDK.1.6_Borland"
Runtime, JRE: 1.1.6_008 (downloaded from your site - SUN ) 
   the problem exist on version 1.2 and 1.3 too.

The problems are:
- it is impossible to unzip a "zip" file that contains a file with a real
  size over 2Gb
- it is impossible to continue with the next "zip" entry if there is a
  problem with the current entry.
- note: I know that it is impossible to do a zip file over 4Gb (but it
  is possible to compress more than 4Gb of data if the result zip file
  has a size less than 4Gb...). It is the norm for eip... But I'm not
  sure this norm says that it is impossible to compress a file between
  2Gb and 4Gb...

The test is:
1) - 6Gb of free space on your disk is needed
   - 3Gb of used space on your disk is needed
   
2) Do a big zip file "big.zip" with 3 files a.dat, b.dat and c.dat :
   - a.dat and c.dat must have a small size (less than 1Mb; 1Kb is fine)
   - b.dat must have a big size (between 2Gb and 4Gb; 2.5Gb is fine. Note:
     1Gb = 1024Mb = 1,048,576Kb = 1073741824 bytes)
   - the name of the 3 files are important (there must be a file before
     and after the big file b.dat in the zip file).
   - for zipping operation, I have download the last version of the
     shareware tools "Winzip" (another zip tools can be used if it is
     compatible with your zip package - I have not try to zip the 3
     files with the zip package in "java.utils" - I'm not sure it would
     work...). Also, I have used this tools to create the b.dat (all my disk
     with no compression).

3) Unzip "big.zip" with the zip package in "java utils".

The results are :
- a.dat is correctly unzipped (if it is not a.dat then it is c.dat; it
  depends on how the file are stored in the "zip" file; so, it depends
  on the tools used for zipping)
- b.dat is not unzipped (a ZipException "invalid entry size" is thrown)
- c.dat can not be unzipped since there was a problem with the previous
  entry b.dat (if it is not c.dat then it is a.dat; ...)
(Review ID: 116880) 
======================================================================

Name: boT120536			Date: 04/22/2001


C:\data\JavaInvestigations\zip>c:\jbuilder3\java\bin\java -version
java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Server VM (build 2.0fcs-E, mixed mode)


I can't seem to write large zip files.

The code below _should_ generate a zip file
with 100,000 entries.
=============================================
import java.util.zip.*;
import java.io.*;


public class ZipTest
{
    private ZipOutputStream zip;

    public static void main (String[] args) throws Exception
    {
        ZipTest t = new ZipTest ();
        t.doIt();
    }

    public void doIt () throws Exception
    {
        zip = new ZipOutputStream (new FileOutputStream ("D:\\temp\\temp.zip"));
        zip.setMethod(ZipOutputStream.STORED);

        for (int i = 0; i < 100000; ++i)
        {
            byte[] data = new byte[1000];

            String entryName = "Fred" + i;

            CRC32 crc32 = new CRC32();
            crc32.update(data, 0, data.length);

            ZipEntry zipentry = new ZipEntry (entryName);
            zipentry.setSize(data.length);
            zipentry.setMethod(ZipEntry.STORED);
    //                    zipentry.setTime();
            zipentry.setCrc(crc32.getValue());
            zip.putNextEntry (zipentry);
            zip.write (data, 0, data.length);
            zip.closeEntry();
        }

        zip.close();
    }
}
========================================================

When I compile it, it runs to completion with no reported
errors.  It generates a file that is _about_ the correct
size (106 MB).  But, the file only reports 34464 entries
when opened.  The last 63000+ entries are not accessible.

This is also a problem with JDK1.2.2
(Review ID: 121141)
======================================================================

Comments
CONVERTED DATA BugTraq+ Release Management Values COMMIT TO FIX: tiger FIXED IN: tiger INTEGRATED IN: tiger tiger-b20
14-06-2004

WORK AROUND Name: boT120536 Date: 02/26/2001 There is no way to bypass this problem whitout rebuilding the zip package in the "java.utils". the problem is in the class "ZipInputStream": private void readEnd(ZipEntry e) throws IOException { ... if (e.size != inf.getTotalOut()) { throw new ZipException( "invalid entry size (expected " + e.size + " but got " + inf.getTotalOut() + " bytes)"); ... } --> e.size is a long on 64 bits (so accepting size over 2Gb) inf.getTotalOut() returns an int on 32 bits (so not accepting size over over 2Gb) --> if all data for the current entry have been read and if an error occured, the attribute "entry" is not set to "null" so it's impossible to get the next zip entry ("getNextEntry" call "closeEntry" that call "read" that call "ReadEnd" that threws an exception for the previous zip entry that causes the error... But if I call "getNextEntry", it is because I want to continue with the next zip entry, and I hope this zip entry is valid...) also : - the native method "GetTotalOut" used in class "Inflater" - the native method "InflatesBytes" used in class "Inflater" They return a signed int on 32 bits (so a value between -2G and +2G). I think they should return a long on 64 bits... ======================================================================
11-06-2004

PUBLIC COMMENTS Add support for zip files and jar files between 2GB and 4GB in size.
10-06-2004

EVALUATION File sizes in zip files are generally stored as unsigned 32-bit quantities. which do not fit into a java `int'. A `long' should have been used instead. For example, public synchronized int getTotalOut() should have been public synchronized long getTotalOut() However, it is too late to change the API, especially for such a "small" fix as to increase the size from 2GB to 4GB. Long term, we want to support file sizes larger than 4GB, and this will require major changes to the Zip File format. PKZIP implements such an extension, and we should investigate their file format. However, we _can_ get support for sizes up to 4GB. Although the return type of getTotalOut and getTotalIn are semantically incorrect, there is no data loss. We simply have to reinterpret the value as unsigned. So it looks like we can fix this without making any API changes. Most of the code in the zip package already uses "long" to represent unsigned 32-bit values, so few changes are necessary. One very good reason to fix this is that currently a user can use the "jar" command to create a jar file that cannot be read by "jar". ###@###.### 2003-04-29
29-04-2003