JDK-8287134 : HttpURLConnection chunked streaming mode doesn't enforce specified size
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.net
  • Affected Version: 8,11,17,18,19
  • Priority: P4
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2022-05-21
  • Updated: 2023-08-08
  • Resolved: 2023-01-26
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 21
21 b08Fixed
Related Reports
CSR :  
Duplicate :  
Relates :  
Description
ADDITIONAL SYSTEM INFORMATION :
java 17
windows 10

A DESCRIPTION OF THE PROBLEM :
[chunkedStreamingMode](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/HttpURLConnection.html#setChunkedStreamingMode(int)) simply enables chunk streaming but it doesn't enforce the specified size on the chunks transmitted. It allows chunks of any size to be transmitted regardless of this value and it doesn't divide the data into multiple chunks if it exceeds this value. We have to do it manually

This method should divide the data in an write() call into chunks automatically without the user manually calculating the size of each chunk but it doesn't.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1) Create an simple server to print the chunks received from the client

2) Create an HttpURLConnection to transfer chunks of size greater than the specified value

3)Observe output on server end. chunk sizes can be of any length irrespective of the length specified in the client method



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Upon transmitting 15 bytes of data and with setChunkedStreamingMode(5) the URL connection should transmit 3 chunks each of size 5 bytes and the last 0 chunk as follows. Output is viewed on the server side

POST /Test.txt HTTP/1.1
User-Agent: Java/17.0.2
Host: localhost:3000
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-type: application/x-www-form-urlencoded
Transfer-Encoding: chunked



=========
5
01234

=========
5
56789

=========
5
ABCDE

=========
0




ACTUAL -
Full 20 bytes is transmitted as one chunk

POST /Test.txt HTTP/1.1
User-Agent: Java/17.0.2
Host: localhost:3000
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-type: application/x-www-form-urlencoded
Transfer-Encoding: chunked


=========
f
0123456789ABCDE

==========
0

---------- BEGIN SOURCE ----------
Run server in an seperate JVM

class Server
{
 public static void main(String[] args)throws Exception
 {
  try(ServerSocket server=new ServerSocket(3000))
  {
   try(Socket client=server.accept())
   {
    try(InputStream input=client.getInputStream())
    {
     int length;
     byte[] data=new byte[6000];
     
     while((length=input.read(data))>0)
     {
      System.out.println(new String(data,0,length));
      System.out.println("=========");
     } 
    } 
   } 
  } 
 }
}

Run client in an seperate JVM

public class Chunked
{
 public static void main(String[] args) throws Exception
 {
  HttpURLConnection con=(HttpURLConnection)new URL("http://localhost:3000/Test.txt")
                        .openConnection();
  
  con.setChunkedStreamingMode(5);
  
  con.setDoOutput(true);
  try(OutputStream output=con.getOutputStream())
  {
   output.write("0123456789ABCDE".getBytes());
   output.flush();
  } 
 }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
in the client we need to manually divide the data our self into equal sized chunks.  

FREQUENCY : always



Comments
Changeset: 48152ef6 Author: Conor Cleary <ccleary@openjdk.org> Date: 2023-01-26 08:35:07 +0000 URL: https://git.openjdk.org/jdk/commit/48152ef66f2466f8c80499325a716de0cb45d8ef
26-01-2023

A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/12137 Date: 2023-01-23 09:04:58 +0000
23-01-2023

Possible update to the documentation could be... "The number of bytes to be written on the wire in each chunk. This includes a chunk size header (1 byte), two CRLF's (4 bytes) and a minimum payload length of 1 byte. If chunklen is less than or equal to 5, a default value will be used."
18-01-2023

So from my understanding and through a little bit of experimentation with submitter code, what Daniel and Daniel have said is correct. When the submitter specifies 5 for the chunk size, that 5 is concerned with the number of bytes sent over the socket in a single write as [~djelinski] pointed out. Seeing as a minimal HTTP chunk also takes up 6 bytes, a value lower than 6 is not usable as previously mentioned. This results in the behavior reported by the submitter. Passing a value of 10 for the chunk length to setChunkedStreamingMode results in the expected behaviour of 5 byte payload lengths. When setChunkedstreamingMode is set to 5, a new ChunkedOutputStream is created where there is logic to use the deafult chunk size value of 4096 instead. So, in conclusion, this comment is just to supplement the explanations previously given. I will update the API documentation accordingly and create a CSR for that when required. Thanks to all for pointing me down the right path!
18-01-2023

Digging up old bugs assigned to me. Will review this presently and create a CSR as needed. Thanks for the previous analysis [~djelinski]. Will update here when required
17-01-2023

Good analysis [~djelinski] - in that case I would advocate for changing the API documentation to document the actual behavior. This would require a CSR.
02-12-2022

The problem is that the behavior of setChunkedStreamingMode does not match its documentation. It treats the chunklen parameter as the number of bytes sent over the socket in a single write. If you want to send 5 byte chunks, you need to use setChunkedStreamingMode(10). The documentation states that: chunklen - The number of bytes to write in each chunk. If chunklen is less than or equal to zero, a default value will be used. In fact the method is using the default chunk size when chunklen is less than or equal to 5. A minimal HTTP chunk of one byte takes up 6 bytes (1 byte for chunk size, 4 bytes for 2 newlines, 1 byte payload), and values lower than 6 are not usable.
02-12-2022

This is related to JDK-6720866. The implementation of ChunkedOutputStream will use a default chunk size if the chunkLength specified is less than the number of bytes needed to encode the chunk header (two CRLF + the chunk size in hexa). This seems wrong. However the crux of the issue seems to be about misaligned buffering of chunks which can lead to performance issues. Understanding the issue reported by JDK-6720866 and the corresponding fix is a pre-requisite to fixing JDK-8287134.
23-05-2022

The observations on Windows 10: JDK 8: Failed, full 20 bytes is transmitted as one chunk JDK 11: Failed. JDK 17: Failed. JDK 18: Failed. JDK 19ea+19: Failed.
23-05-2022