Bug ID: JDK-8229887 (zipfs) zip file corruption when replacing an existing STORED entry

JDK-8229887 : (zipfs) zip file corruption when replacing an existing STORED entry

Type: Bug
Component: core-libs
Sub-Component: java.nio
Affected Version: 11.0.4,12.0.2,13

Priority: P2
Status: Closed
Resolution: Fixed

Submitted: 2019-08-19
Updated: 2024-11-15
Resolved: 2019-08-22

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 11	JDK 13	JDK 14
11.0.24-oracleResolved	13.0.2Fixed	14 b12Fixed

Related Reports

Relates :

JDK-8222532 - (zipfs) Performance regression when writing ZipFileSystem entries in parallel

Sub Tasks

JDK-8230045 :

Release Note: Using the ZIP File System (zipfs) Provider to Update a ZIP or JAR File Containing Uncompressed Entries Might Corrupt the File - Closed

Description

We started noticing corrupted zip files when we tried updating from jdk11.0.3 to jdk11.0.4 and root caused it to this commit

changeset:   54608:c604234be658   
user:        redestad
date:        2019-04-24 15:37 +0200
8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel
Reviewed-by: lancea, clanger, alanb


Here's a self-documenting repro:

 $ make repro
set -x; cat Makefile; cat CorruptZip.java; java --version && echo foobar > foo && jar cM0vf foo.zip foo && javac CorruptZip.java && java CorruptZip; unzip -t foo.zip
+ cat Makefile
# Repro for zip file corruption introduced in 
# 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel
repro:
	set -x; cat Makefile; cat CorruptZip.java; java --version && echo foobar > foo && jar cM0vf foo.zip foo && javac CorruptZip.java && java CorruptZip; unzip -t foo.zip
+ cat CorruptZip.java
import java.nio.file.*;

public class CorruptZip {
    public static void main(String[] args) throws Throwable {
	Path zipPath = Paths.get("foo.zip");
	ClassLoader cl = null;
	try (FileSystem zipfs = FileSystems.newFileSystem(zipPath, cl)) {
	    Files.write(zipfs.getPath("foo"),
			"bar".getBytes("ASCII"),
			StandardOpenOption.TRUNCATE_EXISTING,
			StandardOpenOption.CREATE);
	}
	// Test integrity of zip file:
	new java.util.zip.ZipFile("foo.zip");
    }
}
+ java --version
openjdk 13-internal 2019-09-17
OpenJDK Runtime Environment (build 13-internal+0-adhoc.martinrb.zipfs)
OpenJDK 64-Bit Server VM (build 13-internal+0-adhoc.martinrb.zipfs, mixed mode, sharing)
+ echo foobar
+ jar cM0vf foo.zip foo
adding: foo(in = 7) (out= 7)(stored 0%)
+ javac CorruptZip.java
+ java CorruptZip
Exception in thread "main" java.util.zip.ZipException: invalid END header (bad central directory offset)
	at java.base/java.util.zip.ZipFile$Source.zerror(ZipFile.java:1470)
	at java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1393)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1209)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1172)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:719)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:239)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:169)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:140)
	at CorruptZip.main(CorruptZip.java:14)
+ unzip -t foo.zip
Archive:  foo.zip
error [foo.zip]:  missing 4 bytes in zipfile
  (attempting to process anyway)
error [foo.zip]:  attempt to seek before beginning of zipfile
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)
  (attempting to re-compensate)
foo:  ucsize 3 <> csize 7 for STORED entry
         continuing with "compressed" size value
    testing: foo                      bad CRC 37a79e94  (should be b22c9747)
At least one error was detected in foo.zip.
make: *** [Makefile:4: repro] Error 2

Comments

Fix request (13u, 11u) Requesting this fix to be backported to JDK Updates 11 and 13. It fixes a regression that was caused by JDK-8222532, which got shipped with OpenJDK 11.0.4 and 13. The patch applies cleanly in both, 11u and 13u. However, the included test case uses a new method of java.nio.file.FileSystems that was just introduced with JDK13. Hence it needs a little modification to work for 11. I've sent out a review request: https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2019-September/001838.html It succesfully runs through SAP's regression testing. For JDK11 updates I'm requesting the fix for the October update (11.0.5), since we introduced the regression with 11.0.4. Oracle JDK 11u doesn't seem to be affected. As for JDK13, I'm leaving the call to the maintainers (Oracle) whether it should be promoted to the 13.0.1 update release.
05-09-2019
URL: https://hg.openjdk.java.net/jdk/jdk/rev/abf6ee4c477c User: lancea Date: 2019-08-22 14:44:00 +0000
22-08-2019
This issue is caused by updating a non-compressed entry within a JAR/ZIP file using the ZIP File System (zipfs) provider. The issue has been present in the JDK 13 EA builds since the end of April and was backported to JDK 12.0.2 and OpenJDK 11.0.4. The issue does not occur when updating a compressed entry within a JAR/ZIP file using zipfs, nor does it occur when updating a non-compressed entry with the jar tool or with the java.util.zip API. The default mode for most ZIP tools is to create compressed entries. Therefore the use of non-compressed entries is less common. Given that zipfs is relatively new compared to the jar tool and the java.util.zip API, and that the test suite is not as mature, it would be risky to incorporate the fix at this late stage into JDK 13(GA). Therefore the issue will not be addressed in JDK 13(GA) and will be downgraded to a P2. The fix and tests will be committed to the mainline JDK release (JDK 14). This allows additional time for feedback and provides an opportunity to increase test coverage in this area .
22-08-2019
The following patch addresses the issue. Mach 5 tiers1 - tier3 have been run and are green http://cr.openjdk.java.net/~lancea/8229887/webrev.00/index.html Running Mach5 again with the test case in the patch which ran clean in my workspace Claes has also verified the patch in his workspace
19-08-2019
I "think" the issue surrounds ZipFileSystem.writeEntry() private long writeEntry(Entry e, OutputStream os) throws IOException { if (e.bytes == null && e.file == null) // dir, 0-length data return 0; long written = 0; if (e.csize > 0 && (e.crc != 0 \|\| e.size == 0)) { // pre-compressed entry, write directly to output stream writeTo(e, os); } else { try (OutputStream os2 = (e.method == METHOD_STORED) ? new EntryOutputStreamCRC32(e, os) : new EntryOutputStreamDef(e, os)) { writeTo(e, os2); } } The if block: if (e.csize > 0 && (e.crc != 0 \|\| e.size == 0)) might be the culprit?
19-08-2019
Lance - can you evaluate this quickly to see if this is a blocker issue for JDK 13?
19-08-2019
zipfs was apparently changed to prefer compressing entries to STORing them. If an existing STORED entry is replaced, it looks like the entry is compressed, but the entry's compression method is still bogusly STORED. I would probably choose to preserve the compression method on any pre-existing entries instead of forcing a change from STORED to DEFLATE. The choice of compression seems to be configurable private final int defaultMethod; // METHOD_STORED if "noCompression=true" but that appears to be undocumented.
19-08-2019