United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-6446990 HttpURLConnection#available() reads more and more data into memory
JDK-6446990 : HttpURLConnection#available() reads more and more data into memory

Details
Type:
Bug
Submit Date:
2006-07-08
Status:
Closed
Updated Date:
2011-03-08
Project Name:
JDK
Resolved Date:
2011-03-08
Component:
core-libs
OS:
linux,solaris_10
Sub-Component:
java.net
CPU:
x86
Priority:
P4
Resolution:
Fixed
Affected Versions:
1.4.2_12,5.0
Fixed Versions:

Related Reports
Backport:
Backport:
Backport:
Backport:
Backport:
Duplicate:

Sub Tasks

Description
FULL PRODUCT VERSION :
1.5.0_06 as well as 1.4.2

A DESCRIPTION OF THE PROBLEM :
An input stream from a URLConnection reading a HTTP 1.1 chunked response body reads data into memory when #available() is called. If the client of the URLConnection object calls #available() repeatedly while reading only little data the whole response ends up in memory.

This case occurs in real life if you use a combination of BufferedInputStream, InputStreamReader and BufferedReader to read the response stream line by line as the second testcase demonstrates. It is important to use a buffer size larger  than the default with the BufferedInputStream.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Replace the URL in the testcase with a URL known to point to a large file and known to produce a chunked response. A good way to do this is a servlet that does not set the content length header and simple writes a lot of data.

Run the test case.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The file should be downloaded with minimum heap requirements.

ACTUAL -
You can see the heap increasing, more and more full GCs until you hit OutOfMemory.


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.*;
import java.net.URL;

public class HttpUrlConnectionRunsOutOfMemory {
  public static void main(String[] args) throws IOException {
    InputStream inputStream = new URL("http://localhost:8001/demo/servlet/reportlog?mode=get&name=1151054069229").openStream(); // replace this URL
    BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream, 8192); // default buf size will not trigger the bug
    InputStreamReader inputStreamReader = new InputStreamReader(bufferedInputStream);
    BufferedReader reader = new BufferedReader(inputStreamReader);
    while (reader.readLine() != null);
  }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Use a smaller buf size with the buffered input stream.

                                    

Comments
EVALUATION

Contribution forum : https://jdk-collaboration.dev.java.net/servlets/ProjectForumMessageView?forumID=1463&messageID=14103
                                     
2006-07-08
EVALUATION

Yes, the buffer in the internal ChunkedInputStream implementation should not be let  simply grow without any upper limit. Although not buffering a reasonable amount of data could potentially cause a performance problem for some apps reading large amounts of data, for example downloading an ISO. So the solution here is to set an upper limit for the internal ChunkedInputStream buffer.
                                     
2006-07-11
EVALUATION

Actually setting an upper limit for the internal buffer does not improve performance much. Simply returning the amount currently available in the buffer gives the best performance, and only filling it when it runs out of data.
                                     
2006-07-12
SUGGESTED FIX

..\src\share\classes\sun\net\www\http\ChunkedInputStream.java 

     public synchronized int available() throws IOException {
        ensureOpen();
-       int avail = readAhead(false);
+
+        int avail = chunkCount - chunkPos;
+        if(avail > 0) return avail;
+
+       avail = readAhead(false);
        if (avail < 0) {
            return 0;
        } else  {
                                     
2006-07-12



Hardware and Software, Engineered to Work Together