JDK-8246725 : Provide MAX_ARRAY_SIZE publicly
  • Type: Enhancement
  • Component: core-libs
  • Affected Version: 14
  • Priority: P4
  • Status: Open
  • Resolution: Unresolved
  • Submitted: 2020-06-04
  • Updated: 2024-10-01
Related Reports
Relates :  
JDK-7153247 - Provide public method to insure secure maximum array size
Relates :  
JDK-8298863 - Allow max_jint array length for all element types
Relates :  
JDK-8247373 - ArraysSupport.newLength doc, test, and exception message
Relates :  
JDK-8210577 - adjust OutOfMemoryError specification to mention implementation limits being exceeded
Description
A DESCRIPTION OF THE PROBLEM :
The JVM may throw an OutOfMemoryError if it cannot allocate an array of the requested size, even if there is enough available memory, see JDK-8029587.
The JDK has this maximum size hardcoded at multiple places, but does not expose it publicly. This has the disadvantages that:
- Library authors have to rely on implementation details if they want to prevent OutOfMemoryErrors in their own code
- A third-party JVM might have a lower maximum size than the one assumed by the JDK

Therefore it would be useful to expose this value publicly, possibly as constant of java.lang.reflect.Array (related: JDK-7153247) and have all JVMs adhere to this limit. The limit is currently defined internally by `jdk.internal.util.ArraysSupport.MAX_ARRAY_LENGTH`.
Maybe it would also make sense to differentiate between a guaranteed max array size which all JVM's have to adhere to and the maximum of the currently running JVM (which might be higher), though the difference might be negligible, so maybe such a differentiation is not worth it.
Note however that only exposing the limit of the current JVM and not a general limit would probably cause issues e.g. when a JVM with a higher limit serializes a large array and then a JVM with a lower limit tries to deserialize it.

Additionally the JLS should be updated to mention that an OutOfMemoryError may be thrown if the JVM does not support the requested array size, because apparently this is currently not documented (see https://docs.oracle.com/javase/specs/jls/se14/html/jls-10.html#jls-10.6).
And also refer to the constant field which is proposed by this report.

And JDK classes should then consistently use that constant instead of redeclaring this constant over and over again, examples:
- https://github.com/openjdk/jdk/blob/e1b8e91e80201a62683cc3b5237adc5ae88fa5c7/src/java.base/share/classes/jdk/internal/util/ArraysSupport.java#L609
- https://github.com/openjdk/jdk/blob/e1b8e91e80201a62683cc3b5237adc5ae88fa5c7/src/java.base/share/classes/java/lang/AbstractStringBuilder.java#L238
- https://github.com/openjdk/jdk/blob/e1b8e91e80201a62683cc3b5237adc5ae88fa5c7/src/jdk.zipfs/share/classes/jdk/nio/zipfs/ByteArrayChannel.java#L226
- https://github.com/openjdk/jdk/blob/6bab0f539fba8fb441697846347597b4a0ade428/src/java.base/share/classes/java/util/stream/Nodes.java#L61
- https://github.com/openjdk/jdk/blob/e1b8e91e80201a62683cc3b5237adc5ae88fa5c7/src/java.base/share/classes/java/util/Hashtable.java#L397
- https://github.com/openjdk/jdk/blob/9d764ee48ee7c2e7be7a25aee2ed7bed2fcd2000/src/java.base/share/classes/java/util/ArrayDeque.java#L133
- https://github.com/openjdk/jdk/blob/e1b8e91e80201a62683cc3b5237adc5ae88fa5c7/src/java.base/share/classes/java/util/concurrent/ConcurrentHashMap.java#L520
- https://github.com/openjdk/jdk/blob/e1b8e91e80201a62683cc3b5237adc5ae88fa5c7/src/java.base/share/classes/java/util/concurrent/PriorityBlockingQueue.java#L145
Comments
The REAL fix for this problem is to fix the JVM so that it can allocate arrays of length Integer.MAX_VALUE. From time to time I hear rumors that this might be possible.
01-10-2024
The actual symbol in the JDK has been renamed to SOFT_MAX_ARRAY_LENGTH.
15-04-2022
More information from the submitter with a variety of cases in open source projects where MAX_ARRAY_SIZE (probably copied from the JDK) is used: 1. https://github.com/neo4j/neo4j/blob/a3d854f451021f36845e0df05aee7338ed2088cc/community/common/src/main/java/org/neo4j/internal/helpers/ArrayUtil.java#L34 - Throws RuntimeException - Includes requested or maximum size in exception message - Uses for failing fast for sizes not representable as array length (see IndexedInclusiveLongRange) 2. https://github.com/apache/hive/blob/fecad5b0f72c535ed1c53f2cc62b0d6649b651ae/common/src/java/org/apache/hadoop/hive/common/io/NonSyncByteArrayOutputStream.java#L38 3. https://github.com/apache/hadoop/blob/719b53a79dc169a8c52229831dcb011935a8a151/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/BytesWritable.java#L41 4. https://github.com/apache/accumulo/blob/ef149d41f0d5f00ebd381394ef37bbd2cc21f9b5/core/src/main/java/org/apache/accumulo/core/util/UnsynchronizedBuffer.java#L274 - Creates new arrays, so having the JDK only provide a method for enlarging existing arrays would not be enough 5. https://github.com/imglib/imglib2/blob/b33186acc75abb8292da9865b6d100fc0787644d/src/main/java/net/imglib2/util/Util.java#L81 - Might need to know the limit to choose the respective implementation supporting that many elements 6. https://github.com/apache/kafka/pull/7726 7. https://github.com/google/guava/blob/a1e9a0bd1260a657f7502e5b81d38dfb818cd1d4/guava/src/com/google/common/io/ByteStreams.java#L172 - Uses limit to throw meaningful OutOfMemoryError (including requested size) - Uses limit to fail fast before trying to merge arrays 8. https://github.com/Unidata/netcdf-java/blob/c3e1d83574317eac30066e235942358cc11541b0/cdm/core/src/main/java/ucar/nc2/iosp/hdf5/H5tiledLayoutBB.java#L164 - Uses limit for better exception message and throws IllegalArgumentException instead of OutOfMemoryError - Constructs ByteArrayOutputStream with that size 9. https://github.com/adblockplus/libadblockplus-android/blob/f907e782917db8f6bca7769c3d586bf1c542f0cd/adblock-android/src/main/java/org/adblockplus/libadblockplus/android/Utils.java#L240 (taken from Guava, see above) 10. https://github.com/vigna/fastutil/blob/938d7fafe03960991d0ff4c9da7df6c8fd728e67/src/it/unimi/dsi/fastutil/Arrays.java#L42 11. https://github.com/prestodb/presto/blob/e4d480aabfa21fdbfad4274108f1006661760c87/presto-common/src/main/java/com/facebook/presto/common/block/BlockUtil.java#L34 - Throws IllegalArgumentException with max array size as message - Throws IllegalStateException (there exists another class with the same constant) 12. https://github.com/apache/flink/blob/a3b51f805f247e6aeb681bb36b38da777ba25d79/flink-core/src/main/java/org/apache/flink/util/CollectionUtil.java#L47 (there exist multiple other classes with the same constant) - Throws IllegalStateException with custom message - Throws IllegalArgumentException with message including maximum array length - Uses it for public method (IntValueArray.isFull, DoubleValueArray.isFull, ...) to determine if array can grow 13. https://github.com/airlift/slice/blob/5fd28b8bf56d30b8c22d57ecb1d1f418ec66f975/src/main/java/io/airlift/slice/Slices.java#L40 - Custom exception with message including maximum array length 14. https://github.com/h2oai/h2o-3/blob/a0f4b754cab90e2fc16ef6e6cfdd6e6097ac8fcd/h2o-core/src/main/java/water/AutoBuffer.java#L46 (there exists another class with the same constant) - Throws IllegalArgumentException, IllegalStateException - Creates new arrays, , so having the JDK only provide a method for enlarging existing arrays would not be enough 15. https://github.com/palantir/atlasdb/blob/15a71e977198d275990f8f2b545daf59289cff7a/atlasdb-client/src/main/java/com/palantir/atlasdb/schema/stream/StreamStoreDefinition.java#L34 - Throws IllegalArgumentException with message including maximum array length - Throws IllegalArgumentException with custom message 16. https://github.com/apache/cassandra/blob/7cdad3ce61664cf2ffac75b6e537c910e573754c/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java#L99 - Throws IllegalArgumentException
17-06-2020
We probably wouldn't expose "MAX_ARRAY_SIZE" itself. First, it's an internal library soft limit regarding growable arrays. As such, it's not the actual maximum array size that can be allocated by a JVM. The description contains a useful set of locations that use the soft limit. Ideally these locations should be readjusted to use the central policy implemented in jdk.internal.util.ArraysSupport.newLength. (See JDK-8247373.) The actual maximum array size allowed by a JVM is not a constant across all JVMs, so this would have to be a method, not a constant. That has its own problems, however. (See JDK-7153247.) An alternative approach would be to provide a variation of Arrays.copyOf() that attempts to grow an array (possibly given a minimum growth and a preferred growth, rather like ArraysSupport.newLength), additionally allocating the new array and performing copying as necessary. This could even be handled without copying, if the JVM were to permit it.
13-06-2020
Moved to JDK to discuss the pros and cons of exposing the this maximum size of array.
08-06-2020