JDK-6237956 : mustang b25 cannot extract data from zip files created by third-party zip implementations
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util.jar
  • Affected Version: 6
  • Priority: P1
  • Status: Closed
  • Resolution: Fixed
  • OS: linux,solaris_8,windows_2000
  • CPU: x86,sparc
  • Submitted: 2005-03-09
  • Updated: 2010-12-03
  • Resolved: 2005-04-01
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other JDK 6
1.4.2_13Resolved 6 b31Fixed
Related Reports
Duplicate :  
JDK-6243188 - SAP fails to start from mustang b25
Duplicate :  
JDK-6250871 - REGRESSION: zip file processing is corrupt
Duplicate :  
JDK-6246720 - REGRESSION: ZipException throws when read the contents from .zip files
Relates :  
JDK-6239036 - (cl) classLoader must be able to provide better diagnostics when encountering a malformed JAR
Relates :  
JDK-6236070 - Unexpected eeror in HotSpot VM. It got a SIGBUS.
Relates :  
JDK-6387069 - java -jar z.jar fails when z.jar created using Info-zip
Description
With mustang b25, iMM on S1AS81 can not be accessed.
The error is:
java.lang.NoClassDefFoundError: com.iplanet.ecommerce.vortex.display.PortalServlet
	java.lang.Class.getDeclaredConstructors0(Native Method)
	java.lang.Class.privateGetDeclaredConstructors(Class.java:2328)
	java.lang.Class.getConstructor0(Class.java:2640)
	java.lang.Class.newInstance0(Class.java:321)
	java.lang.Class.newInstance(Class.java:303)
	org.apache.jasper.servlet.JspServletWrapper.getServlet(JspServletWrapper.java:143)
	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:319)
	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:251)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	java.lang.reflect.Method.invoke(Method.java:585)
	org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:249)
	java.security.AccessController.doPrivileged(Native Method)
	javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:282)
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:165)

It's a regression frommustang b24.
In build 25, if we replace libzip.so with b24's libzip.so, the problem is gone.
It's a regression in libzip.so.

###@###.### 2005-03-09 02:37:41 GMT

To reproduce the prooblem:
1. S1AS81 is installed on jtg-e420-5.sfbay ( sparc, sol 8)
   contact ###@###.### for the root passwd of the machine
2. jdk is installed under /usr/j2se
3. execute the script /export/scripts/startS1ASStress.ksh 
4. after the domain starts, access the URL:
   http://jtg-e420-5.sfbay:6060/iMM/iMM/imm.jsp
  with b25/b26, java.lang.NoClassDefFoundError will be displayed on the return
  page.

###@###.### 2005-03-09 07:13:15 GMT
Ran the tests mentioned in the bug and found the fix working.
Comments
SUGGESTED FIX --- /u/martin/ws/mustang/src/share/classes/java/util/zip/ZipFile.java 2005-03-10 16:49:19.226356000 -0800 +++ /u/martin/ws/loc/src/share/classes/java/util/zip/ZipFile.java 2005-03-15 19:08:04.941477000 -0800 @@ -619,10 +619,10 @@ private ByteBuffer directBuffer = null; private String name; - MappedZipFileInputStream(long jzentry, String name) { + MappedZipFileInputStream(long jzentry, String name) throws IOException { super(jzentry); this.name = name; - int offset = (int)getEntryOffset(jzentry); + int offset = (int)getEntryOffset(jzfile, jzentry); MappedByteBuffer bb = ZipFile.this.mappedBuffer; synchronized (bb) { bb.position(offset); @@ -721,7 +721,8 @@ /* If the zip file is mapped, return the offset from the beginning of the zip file to this entry. Return 0 otherwise. */ - private static native long getEntryOffset(long jzentry); + private static native long getEntryOffset(long jzfile, long jzentry) + throws IOException; // Temporary add on for bug troubleshooting private static native String getZipMessage(long jzfile); --- /u/martin/ws/mustang/src/share/native/java/util/zip/ZipFile.c 2004-08-27 16:04:49.370181000 -0700 +++ /u/martin/ws/loc/src/share/native/java/util/zip/ZipFile.c 2005-03-15 19:08:06.265200000 -0800 @@ -197,11 +197,22 @@ } JNIEXPORT jlong JNICALL -Java_java_util_zip_ZipFile_getEntryOffset(JNIEnv *env, jclass cls, jlong zentry) +Java_java_util_zip_ZipFile_getEntryOffset(JNIEnv *env, jclass cls, + jlong zfile, jlong zentry) { #ifdef USE_MMAP jzentry *entry = jlong_to_ptr(zentry); - return entry->pos; + jzfile *zip = jlong_to_ptr(zfile); + jlong offset; + + zip->msg = NULL; + offset = ZIP_GetEntryDataOffset(zip, zentry); + if (offset < 0) { + const char* msg = (zip->msg != NULL) ? zip->msg : + "Error getting zip entry offset"; + ThrowZipException(env, msg); + } + return offset; #else return 0; #endif --- /u/martin/ws/mustang/src/share/native/java/util/zip/zip_util.c 2005-02-09 16:18:41.251642000 -0800 +++ /u/martin/ws/loc/src/share/native/java/util/zip/zip_util.c 2005-03-15 19:08:07.888704000 -0800 @@ -761,7 +761,7 @@ ze->size = CENLEN(cen); ze->csize = (CENHOW(cen) == STORED) ? 0 : CENSIZ(cen); ze->crc = CENCRC(cen); - ze->pos = zip->locpos + CENOFF(cen) + LOCHDR + nlen + elen; + ze->pos = -(zip->locpos + CENOFF(cen)); if ((ze->name = malloc(nlen + 1)) == NULL) goto Catch; memcpy(ze->name, cen + CENHDR, nlen); @@ -945,6 +945,45 @@ MUNLOCK(zip->lock); } +/* + * Returns the offset of the entry data within the zip file. + * Returns -1 if an error occurred, in which case zip->msg will + * contain the error text. + */ +jlong +ZIP_GetEntryDataOffset(jzfile *zip, jzentry *entry) +{ + /* The Zip file spec explicitly allows the LOC extra data size to + * be different from the CEN extra data size, although the JDK + * never creates such zip files. Since we cannot trust the CEN + * extra data size, we need to read the LOC to determine the entry + * data offset. We do this lazily to avoid touching the virtual + * memory page containing the LOC when initializing jzentry + * objects. (This speeds up javac by a factor of 10 when the JDK + * is installed on a very slow filesystem.) + */ + if (entry->pos <= 0) { +#ifdef USE_MMAP + unsigned char *loc = zip->maddr - entry->pos; +#else + unsigned char loc[LOCHDR]; + if (JVM_Lseek(zip->fd, - entry->pos, SEEK_SET) == JVM_IO_ERR) { + zip->msg = "JVM_Lseek failed"; + return -1; + } + if (readFully(zip->fd, loc, LOCHDR) == -1) { + zip->msg = "error reading zip file"; + return -1; + } +#endif + if (GETSIG(loc) != LOCSIG) { + zip->msg = "invalid LOC header (bad signature)"; + return -1; + } + entry->pos = (- entry->pos) + LOCHDR + LOCNAM(loc) + LOCEXT(loc); + } + return entry->pos; +} /* * Reads bytes from the specified zip entry. Assumes that the zip @@ -959,7 +998,7 @@ jlong start; /* Clear previous zip error */ - zip->msg = 0; + zip->msg = NULL; /* Check specified position */ if (pos < 0 || pos > entry_size - 1) { @@ -973,14 +1012,18 @@ if (len > entry_size - pos) len = entry_size - pos; - start = entry->pos + pos; - assert(start >= 0); + /* Get file offset to start reading data */ + start = ZIP_GetEntryDataOffset(zip, entry); + if (start < 0) + return -1; + start += pos; -#ifdef USE_MMAP if (start + len > zip->len) { zip->msg = "ZIP_Read: corrupt zip file: invalid entry size"; return -1; } + +#ifdef USE_MMAP memcpy(buf, zip->maddr + start, len); #else /* Seek to beginning of entry data and read bytes */ @@ -1149,7 +1192,13 @@ /* Clear previous zip error */ zip->msg = 0; - *buf = zip->maddr + entry->pos; + { + jlong offset = ZIP_GetEntryDataOffset(zip, entry); + if (offset < 0) + return JNI_FALSE; + + *buf = zip->maddr + offset; + } ZIP_FreeEntry(zip, entry); --- /u/martin/ws/mustang/src/share/native/java/util/zip/zip_util.h 2005-02-09 16:18:41.406587000 -0800 +++ /u/martin/ws/loc/src/share/native/java/util/zip/zip_util.h 2005-03-15 19:08:08.956574000 -0800 @@ -102,6 +102,9 @@ * - If csize is zero then the entry is uncompressed. * - If extra != 0 then the first two bytes are the length of the extra * data in intel byte order. + * - If pos <= 0 then it is the position of entry LOC header. + * If pos > 0 then it is the position of entry data. + * pos should not be accessed directly, but only by ZIP_GetEntryDataOffset. */ typedef struct jzentry { /* Zip file entry */ @@ -112,7 +115,7 @@ jint crc; /* crc of uncompressed data */ char *comment; /* optional zip file comment */ jbyte *extra; /* optional extra data */ - jlong pos; /* position of entry data */ + jlong pos; /* position of LOC header or entry data */ } jzentry; /* @@ -188,4 +191,6 @@ void ZIP_Unlock(jzfile *zip); jint ZIP_Read(jzfile *zip, jzentry *entry, jlong pos, void *buf, jint len); void ZIP_FreeEntry(jzfile *zip, jzentry *ze); +jlong ZIP_GetEntryDataOffset(jzfile *zip, jzentry *entry); + #endif /* !_ZIP_H_ */ --- /u/martin/ws/mustang/test/java/util/zip/InfoZip.java 1969-12-31 16:00:00.000000000 -0800 +++ /u/martin/ws/loc/test/java/util/zip/InfoZip.java 2005-03-15 19:08:09.651632000 -0800 @@ -0,0 +1,99 @@ +/* @test @(#)InfoZip.java 1.1 05/03/15 + * @bug 6237956 + * @summary We must be able to read zip files created by Unix Info-Zip + * @author Martin Buchholz + */ + +import java.util.*; +import java.util.zip.*; +import java.io.*; + +public class InfoZip { + static int passed = 0, failed = 0; + + static void fail(String msg) { + failed++; + new Exception(msg).printStackTrace(); + } + + static void unexpected(Throwable t) { + failed++; + t.printStackTrace(); + } + + static void check(boolean condition, String msg) { + if (! condition) + fail(msg); + } + + static void check(boolean condition) { + check(condition, "Something's wrong"); + } + + private static String contents(ZipFile zf, ZipEntry ze) throws Exception { + InputStream is = zf.getInputStream(ze); + String result = contents(is); + is.close(); + return result; + } + + private static String contents(InputStream is) throws Exception { + StringBuilder sb = new StringBuilder(); + int c; + while ((c = is.read()) != -1) + sb.append((char)c); + return sb.toString(); + } + + private static void checkZipEntry(ZipEntry ze, String contents) { + check(ze.getName().equals("someFile"), "filename"); + check(ze.getExtra() != null, "extra"); + check(contents.equals("Message in a Bottle\n"), "contents"); + check(ze.getSize() == "Message in a Bottle\n".length()); + } + + + public static void main(String[] args) throws Exception { + //---------------------------------------------------------------- + // The file InfoZip.zip was created using Unix Info-Zip's zip, thus: + // echo Message in a Bottle > someFile; zip InfoZip.zip someFile + // Such a file has a LOC extra different from the CEN extra, + // which cannot happen using JDK APIs. + //---------------------------------------------------------------- + File f = new File("InfoZip.zip"); + + OutputStream os = new FileOutputStream(f); + os.write(new byte[] + {'P', 'K', 3, 4, 10, 0, 0, 0, 0, 0, -68, 8, 'k', + '2', 'V', -7, 'm', 9, 20, 0, 0, 0, 20, 0, 0, 0, + 8, 0, 21, 0, 's', 'o', 'm', 'e', 'F', 'i', 'l', 'e', 'U', + 'T', 9, 0, 3, 't', '_', '1', 'B', 't', '_', '1', 'B', 'U', + 'x', 4, 0, -14, 'v', 26, 4, 'M', 'e', 's', 's', 'a', 'g', + 'e', ' ', 'i', 'n', ' ', 'a', ' ', 'B', 'o', 't', 't', 'l', 'e', + 10, 'P', 'K', 1, 2, 23, 3, 10, 0, 0, 0, 0, 0, + -68, 8, 'k', '2', 'V', -7, 'm', 9, 20, 0, 0, 0, 20, + 0, 0, 0, 8, 0, 13, 0, 0, 0, 0, 0, 1, 0, + 0, 0, -92, -127, 0, 0, 0, 0, 's', 'o', 'm', 'e', 'F', + 'i', 'l', 'e', 'U', 'T', 5, 0, 3, 't', '_', '1', 'B', 'U', + 'x', 0, 0, 'P', 'K', 5, 6, 0, 0, 0, 0, 1, 0, + 1, 0, 'C', 0, 0, 0, 'O', 0, 0, 0, 0, 0, }); + os.close(); + + ZipFile zf = new ZipFile(f); + Enumeration<? extends ZipEntry> entries = zf.entries(); + ZipEntry ze = entries.nextElement(); + check(! entries.hasMoreElements()); + checkZipEntry(ze, contents(zf, ze)); + zf.close(); + + ZipInputStream is = new ZipInputStream(new FileInputStream(f)); + ze = is.getNextEntry(); + checkZipEntry(ze, contents(is)); + check(is.getNextEntry() == null); + + f.delete(); + + System.out.printf("passed = %d, failed = %d%n", passed, failed); + if (failed > 0) throw new Exception("Some tests failed"); + } +} --- /u/martin/ws/mustang/test/java/util/zip/ZipFile/CorruptedZipFiles.java 2005-02-09 16:18:42.275348000 -0800 +++ /u/martin/ws/loc/test/java/util/zip/ZipFile/CorruptedZipFiles.java 2005-03-15 19:08:10.845641000 -0800 @@ -1,5 +1,5 @@ -/* @test @(#)CorruptedZipFiles.java 1.1 05/02/09 - * @bug 4770745 6218846 6218848 +/* @test @(#)CorruptedZipFiles.java 1.2 05/03/15 + * @bug 4770745 6218846 6218848 6237956 * @summary test for correct detection and reporting of corrupted zip files * @author Martin Buchholz */ @@ -42,8 +42,15 @@ int endpos = len - ENDHDR; int cenpos = u16(good, endpos+ENDOFF); + int locpos = u16(good, cenpos+CENOFF); if (u32(good, endpos) != ENDSIG) fail("Where's ENDSIG?"); if (u32(good, cenpos) != CENSIG) fail("Where's CENSIG?"); + if (u32(good, locpos) != LOCSIG) fail("Where's LOCSIG?"); + if (u16(good, locpos+LOCNAM) != u16(good,cenpos+CENNAM)) + fail("Name field length mismatch"); + if (u16(good, locpos+LOCEXT) != u16(good,cenpos+CENEXT)) + fail("Extra field length mismatch"); + byte[] bad; err.println("corrupted ENDSIZ"); @@ -103,19 +110,30 @@ bad[cenpos+CENHOW] = 2; checkZipException(bad, ".*bad compression method.*"); + err.println("corrupted LOCSIG"); + bad = good.clone(); + bad[locpos]++; + checkZipExceptionInGetInputStream(bad, ".*bad signature.*"); + out.printf("passed = %d, failed = %d%n", passed, failed); if (failed > 0) throw new Exception("Some tests failed"); } static int uniquifier = 432; - static void checkZipException(byte[] data, String msgPattern) { + static void checkZipExceptionImpl(byte[] data, + String msgPattern, + boolean getInputStream) { String zipName = "bad" + (uniquifier++) + ".zip"; try { FileOutputStream fos = new FileOutputStream(zipName); fos.write(data); fos.close(); ZipFile zf = new ZipFile(zipName); + if (getInputStream) { + InputStream is = zf.getInputStream(new ZipEntry("x")); + is.read(); + } fail("Failed to throw expected ZipException"); zf.close(); } catch (ZipException e) { @@ -131,6 +149,14 @@ } } + static void checkZipException(byte[] data, String msgPattern) { + checkZipExceptionImpl(data, msgPattern, false); + } + + static void checkZipExceptionInGetInputStream(byte[] data, String msgPattern) { + checkZipExceptionImpl(data, msgPattern, true); + } + static int u8(byte[] data, int offset) { return data[offset]&0xff; } @@ -155,6 +181,16 @@ static final int CENHDR = ZipFile.CENHDR; static final int ENDHDR = ZipFile.ENDHDR; + static final int LOCVER = ZipFile.LOCVER; + static final int LOCFLG = ZipFile.LOCFLG; + static final int LOCHOW = ZipFile.LOCHOW; + static final int LOCTIM = ZipFile.LOCTIM; + static final int LOCCRC = ZipFile.LOCCRC; + static final int LOCSIZ = ZipFile.LOCSIZ; + static final int LOCLEN = ZipFile.LOCLEN; + static final int LOCNAM = ZipFile.LOCNAM; + static final int LOCEXT = ZipFile.LOCEXT; + static final int CENVEM = ZipFile.CENVEM; static final int CENVER = ZipFile.CENVER; static final int CENFLG = ZipFile.CENFLG; ###@###.### 2005-03-16 04:41:39 GMT
12-03-2005
WORK AROUND Unzip the jar file causing problems, and re-zip it using the "jar" command. ###@###.### 2005-03-10 03:05:07 GMT
10-03-2005
EVALUATION As usual, the submitter needs to provide a reproducible test case using only core Java APIs. I know nothing about "S1AS". Is there a jar file which could be opened before, and does not work now? (Probably the one containing the class com.iplanet.ecommerce.vortex.display.PortalServlet) ###@###.### 2005-03-09 18:24:29 GMT Accessing page http://jtg-e420-5:6060/iMM/iMM/include/test.jsp results in "java.lang.NoClassDefFoundError: com.iplanet.ecommerce.vortex.display.PortalServlet" cat test.jsp <%@ page extends="com.iplanet.ecommerce.vortex.display.PortalServlet" %> <% String FILE_PREFIX = "abc"; %> So it looks like The class file "com.iplanet.ecommerce.vortex.display.PortalServlet" can not be accessed. The class file is not in a jar or zip file, it's located under <S1AS root dir>/domains/domain1/applications/j2ee-apps/imm45/imm45_war/WEB-INF/classes/com/iplanet/ecommerce/vortex/display In other words, the class file is placed under <web-application>/WEB-INF/classes/, the path to the class file conforms to web application directory convention. Why the class file can not be found by S1AS needs to be investigated. ###@###.### 2005-03-09 20:40:34 GMT Something seems to be wrong here. There was a large change to libzip introduced in b25, but those changes should only affect access to zip files, not to plain .class files in the file system. If this is indeed a libzip problem, we need to find a zip file that would behave differently with the core APIs in b24 and b25. Use the "jar" commands, and perhaps the ZipFile and ZipInputStream classes to write a small test program. It is known that certain kinds of zip file corruption will cause behavior to be different in b25 vs b24. There should be no change in behavior with valid zip files that worked with b24. ###@###.### 2005-03-09 21:43:41 GMT The problem is understood. The new zip file reading code assumes that the extra data in the LOC and the CEN must be equal, but in fact there is no such requirement. ###@###.### 2005-03-10 02:50:54 GMT /* The Zip file spec explicitly allows the LOC extra data size to * be different from the CEN extra data size, although the JDK * never creates such zip files. Since we cannot trust the CEN * extra data size, we need to read the LOC to determine the entry * data offset. We do this lazily to avoid touching the virtual * memory page containing the LOC when initializing jzentry * objects. (This speeds up javac by a factor of 10 when the JDK * is installed on a very slow filesystem.) */ ###@###.### 2005-03-12 23:12:41 GMT
09-03-2005