JDK-6720866 : Slow performance using HttpURLConnection for upload
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.net
  • Affected Version: 6
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: windows_xp
  • CPU: x86
  • Submitted: 2008-06-30
  • Updated: 2022-05-23
  • Resolved: 2008-12-06
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other Other JDK 6 JDK 7
5.0u17-revFixed 5.0u18Fixed 6u12Fixed 7 b42Fixed
Related Reports
Relates :  
Description
FULL PRODUCT VERSION :
Java(TM) SE Runtime Environment (build 1.6.0_05-b13)
Java HotSpot(TM) Client VM (build 10.0-b19, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
Windows XP

A DESCRIPTION OF THE PROBLEM :
Performance of HttpURLConnecton class for uploading chunked content is significantly slower than using socket and custom implementation of chunked stream

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Performing content upload from client machine to application server (Tomcat).
Use HttpURLConnection to get output stream set on HttpURLConnection method
"POST" and setChunkedStreamingMode(32*1024) after that upload content on the server and record the time it took.
Alternatively we just open socket without any additional configuration and using our implementation of Chunked stream stream the same content on the application server and again measure time.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Stream received from HttpURLConnection should perform not worse than our implementation
ACTUAL -
Stream received from HttpURLConnection actually performs 50-100% worse

ERROR MESSAGES/STACK TRACES THAT OCCUR :
no error messages

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Server side simple servlet that reads from input stream

import java.io.*;
import java.util.Enumeration;
import javax.servlet.ServletException;
import javax.servlet.http.*;

public class TestPostServlet extends HttpServlet
{

    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
        throws ServletException, IOException
    {
        System.out.println((new StringBuilder()).append("DoGet ===> ").append(req.getSession().getId()).toString());
    }

    protected void doPost(HttpServletRequest req, HttpServletResponse resp)
        throws ServletException, IOException
    {
        System.out.println((new StringBuilder()).append("doPost ===> ").append(req.getSession().getId()).toString());
        long start = System.currentTimeMillis();
        InputStream input = req.getInputStream();
        int c;
        do
            c = input.read();
        while(c >= 0);
        long time = System.currentTimeMillis() - start;
        System.out.println((new StringBuilder()).append("time = ").append(time).toString());
    }

    private Object requstId(HttpServletRequest req)
    {
        return req;
    }

    private void dumpHeader(HttpServletRequest req)
    {
        String name;
        for(Enumeration names = req.getHeaderNames(); names.hasMoreElements(); System.out.println((new StringBuilder()).append(name).append(" --> ").append(req.getHeader(name)).toString()))
            name = (String)names.nextElement();

    }

    static long s_date = System.currentTimeMillis();
    private static final int BUFFER_SIZE = 10000;
    private static final int BUFFER_COUNT = 3000;
    private static final String FILENAME = "R:\\__Projects\\AppServer\\out\\exploded\\TestWeb1\\1.ppt";

}

Client side
has two files one is the test:

package com.documentum.test;

import java.io.IOException;
import java.io.OutputStream;
import java.net.*;

public class Test1
{
    public static void main (String[] args)
    {
        try
        {
            new Test1(args[0]).run1();
            Thread.sleep(5000);
            new Test1(args[0]).run2();
            Thread.sleep(5000);
            new Test1(args[0]).run1();
            Thread.sleep(5000);
            new Test1(args[0]).run2();
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }

    public Test1 (String spec) throws MalformedURLException
    {
        m_url = new URL(spec);
    }

    private void run1 () throws IOException
    {
        HttpURLConnection connection = (HttpURLConnection) m_url.openConnection();
        connection.setRequestMethod("POST");
        connection.setDoOutput(true);
        connection.setChunkedStreamingMode(32 * 1024);
        OutputStream out = connection.getOutputStream();

        streamContent(out);

        out.flush ();

        System.out.println("connection.getResponseCode() = " + connection.getResponseCode());

        connection.disconnect();
    }

    private void run2 () throws IOException
    {
        Socket  socket  = new Socket(m_url.getHost(), m_url.getPort());
        OutputStream out = socket.getOutputStream();

        out.write(("POST " +
                   m_url.getPath() +
                   " HTTP/1.1\r\n").getBytes());
        out.write(("host: " +
                   m_url.getHost() +
                   ":" +
                   m_url.getPort() +
                   "\r\n").getBytes());
        out.write("User-agent: TestApp\r\n".getBytes());
        out.write("Transfer-Encoding: chunked\r\n".getBytes());
        out.write("\r\n".getBytes());

        ChunkedOutputStream    chunked = new ChunkedOutputStream(out, 16, 32 * 1024);

        streamContent(chunked);

        chunked.finish ();

        out.close();
        socket.close();
    }

    private static void streamContent (OutputStream out)
        throws IOException
    {
        byte[]  buffer      = new byte[8192];

        for (int i = 0; i < 8192; i++)
        {
            out.write(buffer);
        }
    }

    static final int BUFFER_LENGTH = 32768;
    private URL m_url;
}

Implementation  of chunked output stream
import java.io.OutputStream;
import java.io.IOException;

public class ChunkedOutputStream extends OutputStream
{
    public ChunkedOutputStream (OutputStream stream, int radix)
    {
        this(stream, radix, BUFFER_SIZE);
    }

    public ChunkedOutputStream (OutputStream stream, int radix, int bufferSize)
    {
        m_stream                = stream;
        m_radix                 = radix;
        m_bufferSize            = bufferSize;
        m_standardChunkHeader   = (Integer.toString (m_bufferSize, m_radix) + HEADER_END).getBytes();
        m_fullBufferSize        = m_bufferSize + m_standardChunkHeader.length + FOOTER_BUFFER.length;
        m_buffer                = new byte[m_fullBufferSize];
        m_pos                   = 0;

        initBuffer();
    }

    public void write (int b) throws IOException
    {
        if (m_pos < 0)
            throw new IOException();

        m_buffer[m_standardChunkHeader.length + m_pos++] = (byte)b;

        if (m_pos == m_bufferSize)
            internalFlush(true);
    }

    public void write (byte b[], int off, int len) throws IOException
    {
        if (m_pos < 0)
            throw new IOException();

        int     left    = m_bufferSize - m_pos;

        if (left >= len)
        {
            System.arraycopy(b, off, m_buffer, m_pos + m_standardChunkHeader.length, len);
            m_pos   += len;
            len     = 0;
        }
        else
        {
            System.arraycopy(b, off, m_buffer, m_pos + m_standardChunkHeader.length, left);
            m_pos   += left;
            len     -= left;
        }

        if (m_pos == m_bufferSize)
            internalFlush(false);

        if (len > 0)
            write(b, off + left, len);
    }

    public void flush () throws IOException
    {
        if (m_pos < 0)
            throw new IOException();

        internalFlush(false);
        m_stream.flush ();
        initBuffer();
    }

    public void close () throws IOException
    {
        if (m_pos > -1)
            finish();

        m_stream.close();
    }

    public void finish () throws IOException
    {
        if (m_pos > 0)
            internalFlush(false);

        internalFlush(true);
        m_stream.flush ();

        m_pos   = -1;
    }

    public long getContentLength()
    {
        throw new UnsupportedOperationException(
            "ChunkedOutputStream does not support getContentLength()");
    }

    private void internalFlush (boolean needWriteEmptyChunk) throws IOException
    {
        int offset      = 0;
        int length      = m_fullBufferSize;

        if (m_pos == 0 && !needWriteEmptyChunk)
            return;

        if (m_pos != m_bufferSize)
        {
            byte [] customHeader = Integer.toString (m_pos, m_radix).getBytes();

            offset  = m_standardChunkHeader.length - HEADER_END_BUFFER.length - customHeader.length;

            System.arraycopy(customHeader, 0, m_buffer, offset, customHeader.length);
            System.arraycopy(FOOTER_BUFFER, 0, m_buffer, m_standardChunkHeader.length + m_pos, FOOTER_BUFFER.length);

            length = customHeader.length + FOOTER_BUFFER.length + m_pos + FOOTER_BUFFER.length;
        }

        m_stream.write (m_buffer, offset, length);

        m_pos = 0;
    }

    private void initBuffer ()
    {
        System.arraycopy(m_standardChunkHeader, 0, m_buffer, 0, m_standardChunkHeader.length);
        System.arraycopy(FOOTER_BUFFER, 0, m_buffer, m_buffer.length - FOOTER_BUFFER.length, FOOTER_BUFFER.length);
    }

    private OutputStream m_stream;
    private byte [] m_buffer;
    private int m_pos;
    private final int m_radix;
    private final int m_bufferSize;
    private final int m_fullBufferSize;

    private static final int BUFFER_SIZE = 1460 * 22;

    private final byte[] m_standardChunkHeader;
    private static final byte[] FOOTER_BUFFER       = "\r\n".getBytes ();
    private static final String HEADER_END          = "\r\n";
    private static final byte[] HEADER_END_BUFFER   = HEADER_END.getBytes ();
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Use our own implementation instead of sun HttpURLConnection

Comments
EVALUATION http://hg.openjdk.java.net/jdk7/tl/jdk/rev/a0709a172b6d
26-11-2008

EVALUATION PROBLEM CAUSE: java.net.HttpURLConnection uses sun.net.www.httpChunkedOutputStream which sends data in chunks which size is set with java.net.HttpURLConnection.setChunkedStreamingMode(int size). The problem with the current implementation is that the chunk headers are being buffered in the lower 8k buffer and then if there is a write of larger than 8K the lower buffer needs to be flushed first then write the chunk data, then the footer will be buffered again in the lower 8k buffer, and subsequently flushed. As a result java.io.BufferedOutputStream calls native SocketWrite() more then once to send one data chunk. This has an impact on performance. SOLUTION: Buffering the whole chunk (header,data,footer) in ChunkedOutputStream would allow to write the complete chunk in one go, if it is greater than 8K then it will by pass the lower buffer. FIX DESCRIPTION: The main idea is that write() method collects data in a buffer. When a size of collected (stored) data reaches preferredChunkSize the data (as a complete one chunk) get flushed to an underlying stream in one go. So internal buffer of ChunkedOutputStrem never needs to be re-allaocated to store more then a one data chunk and that's why CR 6526165 and CR 6631048 problem doesn't exist in this version of ChunkedOutputStream.
05-09-2008