United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
JDK-4418997 : Files between 2 Gb and 4Gb (excluded) are not accepted in a zip file.

Details
Type:
Bug
Submit Date:
2001-02-26
Status:
Resolved
Updated Date:
2003-09-12
Project Name:
JDK
Resolved Date:
2003-09-12
Component:
core-libs
OS:
windows_nt
Sub-Component:
java.util.jar
CPU:
x86
Priority:
P4
Resolution:
Fixed
Affected Versions:
1.1.6,1.3.0
Fixed Versions:
5.0 (tiger)

Related Reports
Relates:
Relates:
Relates:
Relates:
Relates:

Sub Tasks

Description

Name: boT120536			Date: 02/26/2001


Development, JDK: "java version JDK.1.6_Borland"
Runtime, JRE: 1.1.6_008 (downloaded from your site - SUN ) 
   the problem exist on version 1.2 and 1.3 too.

The problems are:
- it is impossible to unzip a "zip" file that contains a file with a real
  size over 2Gb
- it is impossible to continue with the next "zip" entry if there is a
  problem with the current entry.
- note: I know that it is impossible to do a zip file over 4Gb (but it
  is possible to compress more than 4Gb of data if the result zip file
  has a size less than 4Gb...). It is the norm for eip... But I'm not
  sure this norm says that it is impossible to compress a file between
  2Gb and 4Gb...

The test is:
1) - 6Gb of free space on your disk is needed
   - 3Gb of used space on your disk is needed
   
2) Do a big zip file "big.zip" with 3 files a.dat, b.dat and c.dat :
   - a.dat and c.dat must have a small size (less than 1Mb; 1Kb is fine)
   - b.dat must have a big size (between 2Gb and 4Gb; 2.5Gb is fine. Note:
     1Gb = 1024Mb = 1,048,576Kb = 1073741824 bytes)
   - the name of the 3 files are important (there must be a file before
     and after the big file b.dat in the zip file).
   - for zipping operation, I have download the last version of the
     shareware tools "Winzip" (another zip tools can be used if it is
     compatible with your zip package - I have not try to zip the 3
     files with the zip package in "java.utils" - I'm not sure it would
     work...). Also, I have used this tools to create the b.dat (all my disk
     with no compression).

3) Unzip "big.zip" with the zip package in "java utils".

The results are :
- a.dat is correctly unzipped (if it is not a.dat then it is c.dat; it
  depends on how the file are stored in the "zip" file; so, it depends
  on the tools used for zipping)
- b.dat is not unzipped (a ZipException "invalid entry size" is thrown)
- c.dat can not be unzipped since there was a problem with the previous
  entry b.dat (if it is not c.dat then it is a.dat; ...)
(Review ID: 116880) 
======================================================================

Name: boT120536			Date: 04/22/2001


C:\data\JavaInvestigations\zip>c:\jbuilder3\java\bin\java -version
java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Server VM (build 2.0fcs-E, mixed mode)


I can't seem to write large zip files.

The code below _should_ generate a zip file
with 100,000 entries.
=============================================
import java.util.zip.*;
import java.io.*;


public class ZipTest
{
    private ZipOutputStream zip;

    public static void main (String[] args) throws Exception
    {
        ZipTest t = new ZipTest ();
        t.doIt();
    }

    public void doIt () throws Exception
    {
        zip = new ZipOutputStream (new FileOutputStream ("D:\\temp\\temp.zip"));
        zip.setMethod(ZipOutputStream.STORED);

        for (int i = 0; i < 100000; ++i)
        {
            byte[] data = new byte[1000];

            String entryName = "Fred" + i;

            CRC32 crc32 = new CRC32();
            crc32.update(data, 0, data.length);

            ZipEntry zipentry = new ZipEntry (entryName);
            zipentry.setSize(data.length);
            zipentry.setMethod(ZipEntry.STORED);
    //                    zipentry.setTime();
            zipentry.setCrc(crc32.getValue());
            zip.putNextEntry (zipentry);
            zip.write (data, 0, data.length);
            zip.closeEntry();
        }

        zip.close();
    }
}
========================================================

When I compile it, it runs to completion with no reported
errors.  It generates a file that is _about_ the correct
size (106 MB).  But, the file only reports 34464 entries
when opened.  The last 63000+ entries are not accessible.

This is also a problem with JDK1.2.2
(Review ID: 121141)
======================================================================

                                    

Comments
CONVERTED DATA

BugTraq+ Release Management Values

COMMIT TO FIX:
tiger

FIXED IN:
tiger

INTEGRATED IN:
tiger
tiger-b20


                                     
2004-06-14
WORK AROUND



Name: boT120536			Date: 02/26/2001


There is no way to bypass this problem whitout rebuilding the zip package
in the "java.utils".

the problem is in the class "ZipInputStream":
 private void readEnd(ZipEntry e) throws IOException {
 ...
   if (e.size != inf.getTotalOut()) {
     throw new ZipException(
       "invalid entry size (expected " + e.size + " but got " +
       inf.getTotalOut() + " bytes)");
 ...
 }

--> e.size is a long on 64 bits (so accepting size over 2Gb)
    inf.getTotalOut() returns an int on 32 bits (so not
    accepting size over over 2Gb)
--> if all data for the current entry have been read and if
    an error occured, the attribute "entry" is not set to
    "null" so it's impossible to get the next zip entry
    ("getNextEntry" call "closeEntry" that call "read" that call
    "ReadEnd" that threws an exception for the previous zip entry
    that causes the error... But if I call "getNextEntry", it is
    because I want to continue with the next zip entry, and I hope
    this zip entry is valid...)

also :
 - the native method "GetTotalOut" used in class "Inflater"
 - the native method "InflatesBytes" used in class "Inflater"
 They return a signed int on 32 bits (so a value between -2G
 and +2G). I think they should return a long on 64 bits...
======================================================================
                                     
2004-06-11
PUBLIC COMMENTS

Add support for zip files and jar files between 2GB and 4GB in size.
                                     
2004-06-10
EVALUATION

File sizes in zip files are generally stored as unsigned 32-bit quantities.
which do not fit into a java `int'.  A `long' should have been used instead.

For example, 
public synchronized int getTotalOut()
should have been
public synchronized long getTotalOut()

However, it is too late to change the API, especially for such a "small"
fix as to increase the size from 2GB to 4GB.

Long term, we want to support file sizes larger than 4GB, and this will
require major changes to the Zip File format.  PKZIP implements such
an extension, and we should investigate their file format.

However, we _can_ get support for sizes up to 4GB.

Although the return type of getTotalOut and getTotalIn are semantically
incorrect, there is no data loss.  We simply have to reinterpret
the value as unsigned.  So it looks like we can fix this without making
any API changes.  Most of the code in the zip package already uses "long"
to represent unsigned 32-bit values, so few changes are necessary.

One very good reason to fix this is that currently a user can use the "jar"
command to create a jar file that cannot be read by "jar".

###@###.### 2003-04-29
                                     
2003-04-29



Hardware and Software, Engineered to Work Together