JDK-8189789 : tomcat gzip-compressed response bodies appear to be broken in update 151
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util.jar
  • Affected Version: 8u151,8u152
  • Priority: P2
  • Status: Resolved
  • Resolution: Fixed
  • OS: generic
  • CPU: x86_64
  • Submitted: 2017-10-22
  • Updated: 2018-07-12
  • Resolved: 2018-04-19
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 10 JDK 7 JDK 8 Other
10Fixed 7u171Fixed 8u162Fixed openjdk7uFixed
Related Reports
Duplicate :  
JDK-8189788 - ERR_CONTENT_DECODING_FAILED in browser using tomcat with compression activated
Duplicate :  
JDK-8200671 - java.util.zip.ZipException: invalid distance too far back in 8u151
Duplicate :  
JDK-8190681 - New zlib 1.2.11 doesn't work in Ubuntu
Duplicate :  
JDK-8190678 - JDK 8 update 151 causes an issue with tomcat gzip compression that did NOT occur using the same version of tomcat and JDK 8 update 144
Relates :  
JDK-8196854 - TestFlushableGZIPOutputStream failing with IndexOutOfBoundsException
Sub Tasks
JDK-8191040 :  
Release Note: Decode error with Tomcat version 7.x - Closed
Description
FULL PRODUCT VERSION :
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Windows 7

A DESCRIPTION OF THE PROBLEM :
After updating to build 151 of the Windows 1.8 JRE, our users all started reporting issues working with our embedded server - which is running the apache-tomcat-7.0.70 webserver.

Disabling compression in the tomcat server.xml or reverting to u144 remedies the issue. Specifically:

java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

REGRESSION.  Last worked in version 8u144

ADDITIONAL REGRESSION INFORMATION: 
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Install JRE 1.8 b151 on a Windows system running Tomcat with "compression" set to "true" for affected connectors.

The response header will indicate that the transfer encoding is "gzip", but will report an error uncompressing the data.

e.g. Chrome returns "ERR_CONTENT_DECODING_FAILED"

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
A valid, decompressed response body
ACTUAL -
Error decompressing the response body (e.g. ERR_CONTENT_DECODING_FAILED)

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Disabling compression in Tomcat (set "compression" to "false" in server.xml)
Comments
Reopening so hgupdater has a chance to close it automatically
19-04-2018
[Regression] test(s) used for verification: java/util/zip/InflateIn_DeflateOut.java JDK 8u151b12 [Windows x64] where the issues has been reproduced: test(s) PASSED JDK 8u171b02 [Windows x64] where the fix is integrated: test(s) PASSED [Regression] test(s) used for verification: GzipTest.java( Bug JDK-8190678) JDK 8u151b12 [Windows x64 + Tomcat 7.0.82] where the issues has been reproduced: test(s) FAILED JDK 8u171b02 [Windows x64 + Tomcat 7.0.82] where the fix is integrated: test(s) PASSED Content-Encoding:gzip Content-Type:text/html;charset=ISO-8859-1 Date:Tue, 23 Jan 2018 08:14:05 GMT Server:Apache-Coyote/1.1 Transfer-Encoding:chunked Vary:Accept-Encoding
23-01-2018
Thanks for test pointers [~phh] Code review started at http://mail.openjdk.java.net/pipermail/core-libs-dev/2017-November/050118.html
22-11-2017
I've attached a reproducer. Does the CPU18_01-defer-approved tag mean that this fix will not be in the January release?
20-11-2017
The tomcat 8.5.23 works fine, the issue is not reproducible. Looked into the tomcat 7.0..82 source code, it appears its gzip handling code has something showed below (to switch the compression level to 0 and flush) to workaround the "flush" issue in our GZIPOutputStream. Tomcat 8 & 9 use the new GZIPOutputStream "flush" flag we introduced in jdk1.7 so they no longer need to switch the compression level. As posted at https://github.com/madler/zlib/issues/305 zlib does have an incompatible change in its deflater code when handling situation someone tries to switch compression level during the compression in its latest releases (since 1.2.9), I attached the details here just for the convenience. public class FlushableGZIPOutputStream extends GZIPOutputStream { ... @Override public synchronized void flush() throws IOException { if (hasLastByte) { // - do not allow the gzip header to be flushed on its own // - do not do anything if there is no data to send // trick the deflater to flush /** * Now this is tricky: We force the Deflater to flush its data by * switching compression level. As yet, a perplexingly simple workaround * for * http://developer.java.sun.com/developer/bugParade/bugs/4255743.html */ if (!def.finished()) { def.setLevel(Deflater.NO_COMPRESSION); flushLastByte(); flagReenableCompression = true; } } out.flush(); } ... } ------------------------------ I took a quick looks at all releases and were trying to have a "unified" workaround. Seems like it's "difficult" to workaround for the releases 1.2.3 and earlier, which does not appears to work well with the "strategy-only-change-in-the-middle-of compression", even tried to use "partial-flush" to replace the z_block. I know these are decade-old releases but they are still showing up on various linux distributions :-( 1.2.3: int ZEXPORT deflateParams(strm, level, strategy) ... if (func != configuration_table[level].func && strm->total_in != 0) { /* Flush the last buffer: */ err = deflate(strm, Z_PARTIAL_FLUSH); } if (s->level != level) { ... } 1.2.4 int ZEXPORT deflateParams(strm, level, strategy) ... if ((strategy != s->strategy || func != configuration_table[level].func) && strm->total_in != 0) { /* Flush the last buffer: */ err = deflate(strm, Z_BLOCK); } if (s->level != level) { ... 1.2.5: int ZEXPORT deflateParams(strm, level, strategy) { ... if ((strategy != s->strategy || func != configuration_table[level].func) && strm->total_in != 0) { /* Flush the last buffer: */ err = deflate(strm, Z_BLOCK); } if (s->level != level) { ... } 1.2.8: int ZEXPORT deflateParams(strm, level, strategy) { ... if ((strategy != s->strategy || func != configuration_table[level].func) && strm->total_in != 0) { /* Flush the last buffer: */ err = deflate(strm, Z_BLOCK); if (err == Z_BUF_ERROR && s->pending == 0) err = Z_OK; } if (s->level != level) { ... } 1.2.9/10: int ZEXPORT deflateParams(strm, level, strategy) { ... if ((strategy != s->strategy || func != configuration_table[level].func)) { /* Flush the last buffer: */ int err = deflate(strm, Z_BLOCK); if (err == Z_STREAM_ERROR) return err; if (strm->avail_out == 0) return Z_BUF_ERROR; } if (s->level != level) { ... } 1.2.11 int ZEXPORT deflateParams(strm, level, strategy) { .... if ((strategy != s->strategy || func != configuration_table[level].func) && s->high_water) { /* Flush the last buffer: */ int err = deflate(strm, Z_BLOCK); if (err == Z_STREAM_ERROR) return err; if (strm->avail_out == 0) return Z_BUF_ERROR; } if (s->level != level) { ... } 1.2.12 (? latest repo) int ZEXPORT deflateParams(strm, level, strategy) { .... if ((strategy != s->strategy || func != configuration_table[level].func) && s->last_flush != -2) { /* Flush the last buffer: */ int err = deflate(strm, Z_BLOCK); if (err == Z_STREAM_ERROR) return err; if (strm->avail_out == 0) return Z_BUF_ERROR; } if (s->level != level) { ... }
07-11-2017
it's reproducible on tomcat 7.0.82 with the test case included in JDKJDK-8190678. And Sean's 8u151/8u152 binary with the patch below appears to make the problem go away. http://cr.openjdk.java.net/~sherman/8189789/webrev
07-11-2017
We (Amazon) have a reproducer, but unfortunately not yet in a form we can publish. We've seen it using tomcat7 and derivatives such as bobcat.
06-11-2017
I can't reproduce it with linux/8u/b151 + tomcat9.0 + gzip enabled. Any version info regarding the tomcat used? I don't think t's a windows only issue, right? ------------------------------------------------------------------------------ [xueshen@sca00kkc tmp]$ ../tomcat/apache-tomcat-9.0.1/bin/catalina.sh run Using CATALINA_BASE: /export/home/xueshen/netty/tomcat/apache-tomcat-9.0.1 Using CATALINA_HOME: /export/home/xueshen/netty/tomcat/apache-tomcat-9.0.1 Using CATALINA_TMPDIR: /export/home/xueshen/netty/tomcat/apache-tomcat-9.0.1/temp Using JRE_HOME: /java/re/jdk/8u151/latest/binaries/linux-x64 Using CLASSPATH: /export/home/xueshen/netty/tomcat/apache-tomcat-9.0.1/bin/bootstrap.jar:/export/home/xueshen/netty/tomcat/apache-tomcat-9.0.1/bin/tomcat-juli.jar 06-Nov-2017 15:09:28.456 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/9.0.1 06-Nov-2017 15:09:28.462 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Sep 27 2017 17:31:52 UTC 06-Nov-2017 15:09:28.463 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 9.0.1.0 06-Nov-2017 15:09:28.463 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux 06-Nov-2017 15:09:28.463 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 3.8.13-68.3.4.el6uek.x86_64 06-Nov-2017 15:09:28.464 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64 06-Nov-2017 15:09:28.464 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /net/scanas416.us.oracle.com/export/java_re/jdk/8u151/fcs/b12/binaries/linux-x64/jre 06-Nov-2017 15:09:28.465 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_151-b12 ----------------------------------------------------------------------------- server.xml: <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" compression="on" /> ------------------------------------------------------------------------------ chrome:devtool: response Headers: Content-Encoding:gzip Content-Type:text/html;charset=ISO-8859-1 Date:Mon, 06 Nov 2017 23:09:33 GMT Transfer-Encoding:chunked Vary:Accept-Encoding ------------------------------------------------------------------------------
06-11-2017
I've changed the OS to "generic" because it also occurs on Amazon Linux.
02-11-2017
See also http://hg.openjdk.java.net/jdk8u/jdk8u/graph/a0672a294b9a.
02-11-2017
This problem is also in 8u152. 8u162-b01 is the version on http://jdk.java.net/8/. It was created *before* 8u151, which latter appears to be the version that contains the switch to a new zlib, which may be what caused the problem. As of now, it seems that tomcat is unusable in all cases where compression is enabled.
02-11-2017
To submitter: Can you please check with JDK 8u162-ea and let us know if you observe the same issue with that , we expect it should be fixed in that version. You can download the same from : http://jdk.java.net/8/
23-10-2017