JDK-4212479 : Data(or Buffered)OutputStream from a URLConnection does not flush writes
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.net
  • Affected Version: 1.1.6
  • Priority: P4
  • Status: Closed
  • Resolution: Not an Issue
  • OS: generic
  • CPU: generic
  • Submitted: 1999-02-18
  • Updated: 2004-08-24
  • Resolved: 1999-05-18
Related Reports
Relates :  
Description

Name: dbT83986			Date: 02/17/99


------------------
When sending big amount of data to a OutputStream from a 
URLConnection (POST method) in chunks of 5k (for example)
OutputStream does not flush, finishing with an OutOfMemory 
Exception.


1. Exact steps to reproduce the problem
   ------------------------------------

   a) Create a new URL object
   b) Obtain its URLConnection
   c) set doOutput and doInput to true
   d) obtain URLConnection's output stream,
   c) an try to send (write) big data in chunks of 5 k. with a loop
   d) Try to flush inside the loop, unsuccessfully.

2. Java SOURCE CODE
   ----------------

import java.io.*;
import java.net.*;

class Prove extends Object
{
  URL u = null;                   // URL of cgi that uploads the file
  URLConnection urlCnx = null;    // URLConnection of previous URL
  FileInputStream fis = null;     // File to send
  String filename = "c:\\tmp\\comm404.exe"; // Size = 27 Mb !!
  public Prove()
  {
    try {
      u = new URL("http://150.1.1.15:8081/cgi-bin/upload.cgi");
      System.out.println("URL = "+u);

      fis = new FileInputStream(filename); 
      System.out.println("FIS = "+fis);

    }catch(Exception e) {
      System.err.println("ERROR in constructor:");
      e.printStackTrace();
    }
  }

  public void doUploadMultipart() 
  {

    try {
      urlCnx = u.openConnection(); // Get Connection
      urlCnx.setDoInput(true);   // RTS: read & write
      urlCnx.setDoOutput(true);  // 
      urlCnx.setUseCaches(false); // We doesn't want cache
      
      
      urlCnx.setRequestProperty("Content-type","multipart/form-data; boundary=sendfile");
    }
    catch(Exception e)
      {
	System.err.println("Error: "+e+"\n");
      }
    
    
    // Now, send the parameters (the file)
    try {
      DataOutputStream printout = new DataOutputStream (urlCnx.getOutputStream());
      
      writeMultipart(printout);
      printout.close();  // Here is were it really flushes
    }catch(IOException e) {
      System.err.println("Error IOException: "+e.getMessage()+"\n");
      e.printStackTrace();
    }

    // We don't reach here, because Exception inside in writeMultipart()
    try
      {
	// Read the result (OK, BAD,...)
	InputStream is = urlCnx.getInputStream();  
	BufferedReader d = new BufferedReader(new InputStreamReader(is));
	String result = d.readLine();
	System.out.println("Result = "+result);
      }
    catch(Exception e)
      {
	System.err.println(e);
	e.printStackTrace();
      }
  }

  protected void writeMultipart(DataOutputStream os)
  {
    try {
      // Write headers
      printBoundaryBegin(os);
      printMPFieldName(os,"file1");
      printMPFileName(os,filename);
      printDoubleReturn(os);
    }catch(IOException error) {
      System.err.println("ERROR: Writing header Multipart");
      System.err.println("MSG = "+error.getMessage());
      System.err.println("E = "+error);
      error.printStackTrace();	      
    }
    
    // Send the file content
    int got;
    byte [] buffer=new byte[1024*5];  // We alloc 5 kb buffer
    
    try {
      while((got=fis.read(buffer)) > 0)  // While data is available, read ...
	{
	  os.write(buffer,0,got);        // ... then write

	  //***************** T H E   P R O B L E M ****************
	  os.flush();                    // ... AND FLUSH (DOES NOT WORK!!!)
	  //***************** T H E   P R O B L E M ****************
	}
    }catch(IOException error){
      System.err.println("ERROR reading & writing file");
      System.err.println("MSG = "+error.getMessage());
      System.err.println("E = "+error);
      error.printStackTrace();
    }
    
    try {
      // Imprime fin de boundary y cierra conexion
      printBoundaryEnd(os);
      os.flush();
    }catch(IOException error) {
      System.err.println("ERROR Writing end of boundary");
      System.err.println("MSG = "+error.getMessage());
      System.err.println("E = "+error);
      error.printStackTrace();
    }
  }

  protected void printBoundaryBegin(OutputStream os) throws IOException
  {
    os.write("--sendfile\r\n".getBytes());    
    os.flush();
  }

  protected void printBoundaryEnd(OutputStream os) throws IOException
  {
    os.write("\r\n--sendfile--\r\n".getBytes());    
    os.flush();
  }

  protected void printMPFieldName(OutputStream os,String fieldname) throws IOException
  {
    os.write(("Content-Disposition: form-data; name=\""+fieldname+"\"").getBytes());
    os.flush();
  }

  protected void printMPFileName(OutputStream os, String filename) throws IOException
  {
    os.write(("; filename=\""+filename+"\"").getBytes());
    os.flush();
  }

  protected void printDoubleReturn(OutputStream os) throws IOException
  {
    os.write("\r\n\r\n".getBytes());
    os.flush();
  }
  
  public static void main(String argv[])
  {
    Prove p = new Prove();

    p.doUploadMultipart();
  }
}

3. Text Error Message
   ------------------

java.lang.OutOfMemoryError: 

4. Trace info
   ----------

java.lang.OutOfMemoryError: 
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:95)
	at java.io.DataOutputStream.write(DataOutputStream.java:83)
	at Prove.writeMultipart(Prove.java:92)
	at Prove.doUploadMultipart(Prove.java:47)
	at Prove.main(Prove.java:151)

5. java -fullversion
   -----------------

  java full version "JDK1.1.6N"

6. Note
   ----

   a) I have replaced the DataOutputStream with an BufferedOutputStream 
      with the same results.
   b) CGI's hangs because header content-length noticed it to 
      send more bytes than it really can obtain. So, it hangs
      waiting for read more bytes at fread in stdin.
(Review ID: 54262)
======================================================================

Comments
EVALUATION The getOutputStream method of sun.net.www.protocol.http.HttpURLConnection returns a ByteArrayOutputStream, so if you try to write 27MB into it you're likely to get an OutOfMemoryError. Reassigning to classes_net. -- mr@eng 1999/5/18 HTTP 1.0 requires sending a Content-Length request header to identify the length of a request such as POST which has a variable length request "message body". So, HttpURLConnection returns a ByteArrayOutputStream from getOutputStream() in order to buffer up and count the size of the data to be posted. Large amounts of post data require large amounts of buffering. The customer should see if they can modify their single large POST into multiple smaller POSTs. Or, increase the maximum JVM address space to avoid the out of memory exception. This issue will be largely mitigated when we add HTTP 1.1 client support. This will allow us to use "chunked" Transfer-Encoding when talking to known HTTP 1.1 servers. This allows us to use a fixed-size buffer when calculating the transfer fix rather than having to buffer up the entire request all at once. However, this trick only works when talking to servers that we have already determined to be HTTP 1.1 servers. I'm closing this as "not a bug" since this is not inherently a bug and just a protocol limitation, the real problem is that the JVM really did run out of it's maximum heap size, the description is not really accurate, and this problem cannot be "fixed" if the server doesn't support HTTP 1.1. There is already an RFE, 4116886, asking for HTTP 1.1 support. jeff.nisewanger@Eng 1999-05-18 Although the bug was closed to be tracked by a RFE 4116886, the underlying problem was not fixed until Tiger, with a new API to support Http request streaming (new API is needed, because backward compatibility needs to be preserved for the current behaviour). See bug 5026745. Also, this new enhancement will be documented on the Tiger(5.0) networking features and enhancement section of the 5.0 release notes. ###@###.### 2004-08-24
24-08-2004

WORK AROUND Increase the maximum Java heap size via the -mx or -Xmx command line flags. jeff.nisewanger@Eng 1999-05-18
18-05-1999