JDK-4705373 : new java.util.zip.ZipFile causes OutOfMemoryError for large zip files
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util.jar
  • Affected Version: 1.4.0
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • OS: linux
  • CPU: x86
  • Submitted: 2002-06-20
  • Updated: 2005-12-15
  • Resolved: 2005-12-15
Related Reports
Duplicate :  
Description
Name: nt126004			Date: 06/20/2002


FULL PRODUCT VERSION :
>java -version
java version "1.4.0"
Java(TM) 2 Runtime Environment, Standard Edition(build 1.4.0-b92)
Java HotSpot(TM) Client VM (build 1.4.0-b92, mixed mode)

vm params:
-
server -Xmaxf.35 -Xminf.15 -Xmx600m

FULL OPERATING SYSTEM VERSION :
os: linux
Distribution:
redhat
>uname -a
Linux XXXX 2.4.18 #1 SMP Wed Apr 24 13:41:20 CEST 2002
i686 unknown
>rpm --query glibc
glibc-2.1.2-11

ADDITIONAL
OPERATING SYSTEMS :



A DESCRIPTION OF THE PROBLEM :
Opening a large zip file (larger than specified -Xmx size) with

   new
java.util.zip.ZipFile("large.zip")

causes an
OutOfMemoryError.

It appears that the jvm is trying to mmap the entire file.

Here is additional info from "strace -f java ...":

The memory allocations fails in Sun 1.4 at

open("test3.zip", O_RDONLY|O_LARGEFILE) = 5
fstat64(5, {st_mode=S_IFREG|0644, st_size=2037717101, ...}) = 0
_llseek(5, 0, [2037717101], SEEK_END) = 0
old_mmap(NULL, 2037717101, PROT_READ, MAP_SHARED, 5, 0) = -1 ENOMEM (Cannot allocate memory)
close(5) = 0
write(2, "java.lang.OutOfMemoryError", 26

The bytecount there is the size of my zipfile and I have no idea, why
the Java-VM calls mmap(size_of_zipfile) there, which means it can only
handle zipfiles which fit into system-memory.

This was tested with different Java-VMs and a ZIP-file of 1.9 GB size.
The results:

Sun 1.3: works
Sun 1.3.1 (release 2): fails, but works with -classic Option
Sun 1.4.0: fails (no -classic option available)

i.b.m.'s jre 1.3: works

Blackdown 1.3: works
Blackdown 1.3.1: fails, but works with -classic

The call used was without any options: java LargeZipFile test.zip

System:
Pentium III
Mem:    256 M
Swap:   512 M

Debian Linux (woody)
Kernel 2.4.18
libc-2.2.5
libpthread-0.9




REGRESSION.  Last worked in version 1.3.1

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. create a large zip file
2. start test program


EXPECTED VERSUS ACTUAL BEHAVIOR :
expected:
    instantiation of a new ZipFile object
actual result:
    
OutOfMemoryError

ERROR MESSAGES/STACK TRACES THAT OCCUR :
>java -server -Xmaxf.35 -Xminf.15 -Xmx600m LargeZipFile large.zip
totoal memory:
2031616
free memory: 1769520
large.zip, size is 1412020128
bytes
java.lang.OutOfMemoryError
        at java.util.zip.ZipFile.open(Native Method)
        at
java.util.zip.ZipFile.<init>(ZipFile.java:112)
        at
java.util.zip.ZipFile.<init>(ZipFile.java:72)
        at
LargeZipFile.main(LargeZipFile.java:7)

This bug can be reproduced always.

---------- BEGIN SOURCE ----------
public class LargeZipFile {
	public static void main(String[] args) {
		try {
			System.out.println("totoal memory: " +
Runtime.getRuntime().totalMemory());
			System.out.println("free memory: " + Runtime.getRuntime().freeMemory());
			System.out.println(args[0] + ", size is " + new
java.io.File(args[0]).length() + " bytes");
			new java.util.zip.ZipFile(args[0]);
			System.out.println("success");
		} catch (Throwable t) {
			t.printStackTrace();
		}
	}
}
---------- END SOURCE ----------

CUSTOMER WORKAROUND :
No workaround possible.
(Review ID: 145979) 
======================================================================

Comments
EVALUATION I believe that the fix for 6280693: Mmap the whole jar files takes too much perceived footprint has made this problem almost disappear (or at least, for typical zip files, a factor of 50 less memory is being mmapped). This, together with the fact that very large zip files (larger than 4GB) are not supported, means that this problem is effectively fixed. Of course, my claim should be tested. If confirmed, this bug should be closed as Not Reproducible.
11-11-2005