JDK-8206403 : ByteArrayOutputStream hugeCapacity method can return invalid capacity
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.io
  • Affected Version: 8,9,10,11
  • Priority: P4
  • Status: Resolved
  • Resolution: Won't Fix
  • OS: generic
  • CPU: generic
  • Submitted: 2018-07-02
  • Updated: 2018-07-24
  • Resolved: 2018-07-24
Related Reports
Relates :  
Description
A DESCRIPTION OF THE PROBLEM :
private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
    
private static int hugeCapacity(int minCapacity) {
      if (minCapacity < 0) // overflow
          throw new OutOfMemoryError();
      return (minCapacity > MAX_ARRAY_SIZE) ?
          Integer.MAX_VALUE :
          MAX_ARRAY_SIZE;

The maximum size of an array the VM lets create is Integer.MAX_VALUE - 2. (2147483645)
The MAX_ARRAY_SIZE constant's value is Integer.MAX_VALUE - 8. (2147483639)

So any value between the two will result in hugeCapacity function to return Integer.MAX_VALUE.
Which value is too high so it will cause an OutOfMemoryError.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
    byte[] buf = new byte[Integer.MAX_VALUE-6];
    ByteArrayOutputStream out = new ByteArrayOutputStream(1);
    out.write(buf);

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The underlaying array in the ByteArrayOutputStream can be the same size as buff variable.
ACTUAL -
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at java.io.OutputStream.write(OutputStream.java:75)
	at com.company.Main.main(Main.java:17)

---------- BEGIN SOURCE ----------
package com.company;

import java.io.ByteArrayOutputStream;
import java.io.IOException;

public class Main {
  public static void main(String[] args) throws IOException {
    byte[] buf = new byte[Integer.MAX_VALUE-6];
    ByteArrayOutputStream out = new ByteArrayOutputStream(1);
    out.write(buf);
  }
}
---------- END SOURCE ----------

FREQUENCY : always



Comments
The test case is hitting an implementation limit, not a bug. Please refer to JDK-8055949 for context on why the implementation is as it is. Resolving this as Will Not Fix for several reasons: 1) The number of bytes used for a VM's header of an array might vary by implementation but 8 seems to be a safe value, i.e., general upper bound. 2) It is probably not worth the effort to try to obtain the last few bytes before throwing an OOME. 3) The MAX_ARRAY_SIZE constant is used in a number of different classes which should all be reexamined and possibly modified for consistency if the change described in this issue were applied.
24-07-2018

Why would it even permit creating an array greater than MAX_ARRAY_SIZE?
20-07-2018

Question is why does it pass Integer.MAX_VALUE when the requested amount is greater than MAX_ARRAY_SIZE. It might be better to just return the requested amount, which would be a slightly more conservative approach. diff -r 995f511e5530 src/java.base/share/classes/java/io/ByteArrayOutputStream.java --- a/src/java.base/share/classes/java/io/ByteArrayOutputStream.java Tue Jul 03 14:14:17 2018 +0100 +++ b/src/java.base/share/classes/java/io/ByteArrayOutputStream.java Thu Jul 05 12:12:01 2018 +0100 @@ -124,7 +124,7 @@ if (minCapacity < 0) // overflow throw new OutOfMemoryError(); return (minCapacity > MAX_ARRAY_SIZE) ? - Integer.MAX_VALUE : + minCapacity : MAX_ARRAY_SIZE; }
05-07-2018

Integer.MAX_VALUE minus 8 has been a safe limit in the face of different VM builds/variants and object/memory layouts (-XX:-UseCompressedOops etc.). Changing this would require investigation for what seems like a corner case.
05-07-2018

To reproduce the issue, run the attached test case. JDK 8u171 - Fail JDK 10.0.1 - Fail JDK 11-ea+18- Fail Output : Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.base/java.util.Arrays.copyOf(Arrays.java:3745) at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120) at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95) at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156) at java.base/java.io.OutputStream.write(OutputStream.java:122) at JI9055869.main(JI9055869.java:9)
05-07-2018