JDK-6631048 : Problem when writing on output stream of HttpURLConnection
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.net
  • Affected Version: 6
  • Priority: P4
  • Status: Closed
  • Resolution: Fixed
  • OS: windows_xp
  • CPU: x86
  • Submitted: 2007-11-16
  • Updated: 2011-03-08
  • Resolved: 2011-03-08
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 6 JDK 7
6u10Fixed 7 b25Fixed
Related Reports
Relates :  
Description
FULL PRODUCT VERSION :
java version "1.6.0_05-ea"
Java(TM) SE Runtime Environment (build 1.6.0_05-ea-b06)
Java HotSpot(TM) Client VM (build 1.6.0_05-ea-b06, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

A DESCRIPTION OF THE PROBLEM :
when using the HttpURLConnection's output stream to write data we are facing problem. The chunking size is set to 48K.  This piece of code works well in jdk 5. but the same code fails for jdk 6. Looking into the source code, there seems a 10K  threshold limit is introduced in jdk 6. This piece of code is creating problem.

I have attached a test program to reproduce the problem. The program works fine for JDK 5 without any problem. The exception stack trace is:

java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at sun.net.www.http.ChunkedOutputStream.write(ChunkedOutputStream.java:161)
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:2492)
	at com.documentum.ucf.common.transport.impl.TestHttpURLConnection.writeData(TestHttpURLConnection.java:63)
	at com.documentum.ucf.common.transport.impl.TestHttpURLConnection.testHttpUrlConnectionWrite(TestHttpURLConnection.java:45)
	at com.documentum.ucf.common.transport.impl.TestHttpURLConnection.main(TestHttpURLConnection.java:26)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. compile the source
2. Run it with jdk 1.5 - you wont see any problem
3. Run it with jdk 6 - you will find the problem.

You can connect to a Test servlet, in which case also you can see the problem.


ERROR MESSAGES/STACK TRACES THAT OCCUR :
C:\Trials\Test\src>java TestHttpURLConnection
Writing buf len, len: 32120,32120
java.lang.ArrayIndexOutOfBoundsException
        at java.lang.System.arraycopy(Native Method)
        at sun.net.www.http.ChunkedOutputStream.write(Unknown Source)
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(Unknown Source)
        at TestHttpURLConnection.writeData(TestHttpURLConnection.java:60)
        at TestHttpURLConnection.testHttpUrlConnectionWrite(TestHttpURLConnection.java:42)
        at TestHttpURLConnection.main(TestHttpURLConnection.java:24)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;

public class TestHttpURLConnection
{

    public static void main (String[] args)
    {
        TestHttpURLConnection testHttpURLConnection = new TestHttpURLConnection();
        try
        {
            testHttpURLConnection.testHttpUrlConnectionWrite();
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }

    public void testHttpUrlConnectionWrite () throws Exception
    {
        //Test - If you want to , change it to the url of a test servlet in any app server, the problem will be there even otherwise.
        URL url = new URL("http://some_host:8080");

        HttpURLConnection urlConnection = open(url);

        setChunkingMode(urlConnection, CHUNK_SIZE);

        OutputStream out = urlConnection.getOutputStream();

        writeData(out);

        out.close();
    }

    private void writeData (OutputStream out) throws IOException
    {
        //you can replace this with any file or simply a byte[] data
        String file = "C:\\tmp\\test_1m.pdf";

        InputStream is = new FileInputStream(file);

        byte[] buf = new byte[BUFFER_SIZE];
        int len = 0;

        while ((len = is.read(buf)) >= 0)
        {
            System.out.println("Writing buf len, len: " + buf.length + "," + len);
            out.write(buf, 0, len);
        }

        is.close();
    }

    private HttpURLConnection open (URL url) throws IOException
    {
        HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
        urlConnection.setDoInput(true);
        urlConnection.setDoOutput(true);
        return urlConnection;
    }

    private void setChunkingMode (HttpURLConnection urlConnection, int chunkSize) throws IOException
    {
        urlConnection.setChunkedStreamingMode(chunkSize);
    }

    /**
       * Test comments - I played with the BUFFER_SIZE and CHUNK_SIZE values, but that didn't fix the problem.
       */
    private static final int BUFFER_SIZE = 32120;

    private static final int CHUNK_SIZE = 48 * 1024;

}


---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
I coudn't find anything. By changing the buffer size and chunk size, you may reduce the problem, but happens though.

Comments
WORK AROUND The workaround is to write to the output stream in chunks less than 10K. For example: os.write (buf, 0, (10 * 1024)); os.write (buf, (10 * 1024), (10 * 1024)); os.write (buf, (20 * 1024), (10 * 1024));
17-12-2007

EVALUATION I don't think that the IOException being thrown from 1.5 is a problem. This just indicates that the server is not accepting the data or has closed the connection. This could be the case if the server is not setup correctly. I have tested against a valid server and the test program completes successfully with 1.5 and throws ArrayIndexOutOfBoundsException with 6.0. This appears to be a regression introduced from the fix for 6526165. The fix for 6526165 causes byte array writes greater than 10K ( 10 * 1024 bytes ) to potentially by pass the internal buffering of ChunkedOutputStream and write directly to the underlying stream. The reason this fails is that in the case described in the description section, the chunk size is greater than the byte array being written, and both are greater than 10K. The internal logic of ChunkedOutputStream.write does not expect this. The fix is simply to determine if the amount of data being written is greater than both 10K and what is remaining in the current chunk. Otherwise simply write it to the internal buffer.
17-12-2007

EVALUATION I ran the testcase on jdk5.0u13 and jdk1.6.0_03 and here is the output I see: jdk1.5.0_13 output: ------------------- Writing buf len, len: 32120,32120 Writing buf len, len: 32120,32120 Writing buf len, len: 32120,32120 java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.che ckError(HttpURLConnection.java:2285) at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.wri te(HttpURLConnection.java:2268) at TestHttpURLConnection.writeData(TestHttpURLConnection.java:54) at TestHttpURLConnection.testHttpUrlConnectionWrite(TestHttpURLConnectio n.java:36) at TestHttpURLConnection.main(TestHttpURLConnection.java:17) jdk1.6.0_03 output: ------------------- Writing buf len, len: 32120,32120 java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at sun.net.www.http.ChunkedOutputStream.write(ChunkedOutputStream.java:1 61) at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.wri te(HttpURLConnection.java:2513) at TestHttpURLConnection.writeData(TestHttpURLConnection.java:54) at TestHttpURLConnection.testHttpUrlConnectionWrite(TestHttpURLConnectio n.java:36) at TestHttpURLConnection.main(TestHttpURLConnection.java:17) The submitter has claimed that the testcase runs fine on 5.0 which is not what I observe here. Could you please clarify and let me know what version of 5.0 did you try out with?
17-12-2007