JDK-8133170 : Deflater.deflate with small output buffers fails
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util.jar
  • Affected Version: 8u45,9
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2015-08-06
  • Updated: 2016-07-15
  • Resolved: 2016-07-07
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 9
9 b127Fixed
Description
FULL PRODUCT VERSION :
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux lapetus 3.19.8-100.fc20.x86_64 #1 SMP Tue May 12 17:08:50 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
Originally asked as question on stackoverflow
http://stackoverflow.com/questions/31861983/deflater-deflate-and-small-output-buffers

Copied here (I am author of original stackoverflow question)

I'm seeing a strange situation with small output buffers with Java 8u45 and the java.util.Deflater.deflate(byte[] b, int off, int len, int flush) method when used with small output buffers.

(I'm working on some low level networking code related to WebSocket's upcoming permessage-deflate extension, so small buffers are a reality for me)

If a buffer of 5 or 6 bytes is used as an output buffer for .deflate() with SYNC_FLUSH then there is no way for the deflate to finish.



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run test class

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
In the provided example class, I'm attempting to generate compressed bytes for the input "Hello" using an output buffer of 5 bytes in length.

3 buffers through use of example code attempting to compress "Hello" (5 bytes) using an output buffer of 5 bytes.

buffer 1 [ F2 48 CD C9 C9 ]
buffer 2 [ 07 00 00 00 FF ]
buffer 3 [ FF ]

Which translates as

[ F2 48 CD C9 C9 07 00 ] <-- the compressed data
[ 00 00 FF FF ]          <-- the deflate tail bytes

Other possible outcome :

A RuntimeException on use of small output buffers sizes when using SYNC_FLUSH with .deflate()


ACTUAL -
Infinite loop

Resulting output of running the provided demo code ...

input is 5 bytes - [48 65 6C 6C 6F]
compressed 5 bytes - [F2 48 CD C9 C9]
compressed 5 bytes - [07 00 00 00 FF]
compressed 5 bytes - [FF 00 00 00 FF]
compressed 5 bytes - [FF 00 00 00 FF]
compressed 5 bytes - [FF 00 00 00 FF]
compressed 5 bytes - [FF 00 00 00 FF]
compressed 5 bytes - [FF 00 00 00 FF]
compressed 5 bytes - [FF 00 00 00 FF]
compressed 5 bytes - [FF 00 00 00 FF]
compressed 5 bytes - [FF 00 00 00 FF]
Exited compress (maxloops left -1)



REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
package deflate;

import java.nio.charset.StandardCharsets;
import java.util.zip.Deflater;

public class DeflaterSmallBufferBug
{
    public static void main(String[] args)
    {
        boolean nowrap = true;
        Deflater deflater = new Deflater(Deflater.DEFAULT_COMPRESSION,nowrap);

        byte[] input = "Hello".getBytes(StandardCharsets.UTF_8);

        System.out.printf("input is %,d bytes - %s%n",input.length,getHex(input,0,input.length));

        deflater.setInput(input);

        byte[] output = new byte[input.length];

        // break out of infinite loop seen with bug
        int maxloops = 10;

        // Compress the data
        while (maxloops-- > 0)
        {
            int compressed = deflater.deflate(output,0,output.length,Deflater.SYNC_FLUSH);
            System.out.printf("compressed %,d bytes - %s%n",compressed,getHex(output,0,compressed));

            if (compressed < output.length)
            {
                System.out.printf("Compress success");
                return;
            }
        }

        System.out.printf("Exited compress (maxloops left %d)%n",maxloops);
    }

    private static String getHex(byte[] buf, int offset, int len)
    {
        StringBuilder hex = new StringBuilder();
        hex.append('[');
        for (int i = offset; i < (offset + len); i++)
        {
            if (i > offset)
            {
                hex.append(' ');
            }
            hex.append(String.format("%02X",buf[i]));
        }
        hex.append(']');
        return hex.toString();
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
After asking the question on stackoverflow, Mark Adler (co-author of zlib/gzip) answered and pointed out that there is a (greater than) 6 byte output buffer requirement for use of SYNC_FLUSH successfully with zlib.




Comments
1. Run the attached test case in Windows 7. 2. Checked this for JDK 8u51, 8u60 ea b26, 9 ea b75 8u40: FAIL 8u51: FAIL 8u60 ea b26: FAIL 9 ea b75: FAIL 3. Output with JDK 8u51: > java DeflaterSmallBufferBug input is 5 bytes - [48 65 6C 6C 6F] compressed 5 bytes - [F2 48 CD C9 C9] compressed 5 bytes - [07 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] compressed 5 bytes - [FF 00 00 00 FF] Exited compress (maxloops left -1) 4. This issue is reproducible with JDK 8u51, 8u60 ea b26 and 9 ea b75.
07-08-2015