JDK-6446990 : HttpURLConnection#available() reads more and more data into memory
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.net
  • Affected Version: 1.4.2_12,5.0
  • Priority: P4
  • Status: Closed
  • Resolution: Fixed
  • OS: linux,solaris_10
  • CPU: x86
  • Submitted: 2006-07-08
  • Updated: 2011-03-08
  • Resolved: 2011-03-08
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other Other Other JDK 6 JDK 7
1.4.2_18-revFixed 1.4.2_19Fixed 5.0u17Fixed 6u10Fixed 7 b03Fixed
Related Reports
Duplicate :  
Description
FULL PRODUCT VERSION :
1.5.0_06 as well as 1.4.2

A DESCRIPTION OF THE PROBLEM :
An input stream from a URLConnection reading a HTTP 1.1 chunked response body reads data into memory when #available() is called. If the client of the URLConnection object calls #available() repeatedly while reading only little data the whole response ends up in memory.

This case occurs in real life if you use a combination of BufferedInputStream, InputStreamReader and BufferedReader to read the response stream line by line as the second testcase demonstrates. It is important to use a buffer size larger  than the default with the BufferedInputStream.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Replace the URL in the testcase with a URL known to point to a large file and known to produce a chunked response. A good way to do this is a servlet that does not set the content length header and simple writes a lot of data.

Run the test case.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The file should be downloaded with minimum heap requirements.

ACTUAL -
You can see the heap increasing, more and more full GCs until you hit OutOfMemory.


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.*;
import java.net.URL;

public class HttpUrlConnectionRunsOutOfMemory {
  public static void main(String[] args) throws IOException {
    InputStream inputStream = new URL("http://localhost:8001/demo/servlet/reportlog?mode=get&name=1151054069229").openStream(); // replace this URL
    BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream, 8192); // default buf size will not trigger the bug
    InputStreamReader inputStreamReader = new InputStreamReader(bufferedInputStream);
    BufferedReader reader = new BufferedReader(inputStreamReader);
    while (reader.readLine() != null);
  }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Use a smaller buf size with the buffered input stream.

Comments
EVALUATION Actually setting an upper limit for the internal buffer does not improve performance much. Simply returning the amount currently available in the buffer gives the best performance, and only filling it when it runs out of data.
12-07-2006

SUGGESTED FIX ..\src\share\classes\sun\net\www\http\ChunkedInputStream.java public synchronized int available() throws IOException { ensureOpen(); - int avail = readAhead(false); + + int avail = chunkCount - chunkPos; + if(avail > 0) return avail; + + avail = readAhead(false); if (avail < 0) { return 0; } else {
12-07-2006

EVALUATION Yes, the buffer in the internal ChunkedInputStream implementation should not be let simply grow without any upper limit. Although not buffering a reasonable amount of data could potentially cause a performance problem for some apps reading large amounts of data, for example downloading an ISO. So the solution here is to set an upper limit for the internal ChunkedInputStream buffer.
11-07-2006

EVALUATION Contribution forum : https://jdk-collaboration.dev.java.net/servlets/ProjectForumMessageView?forumID=1463&messageID=14103
08-07-2006