JDK-6478546 : FileInputStream.read() throws OutOfMemoryError when there is plenty available
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.io
  • Affected Version: 6
  • Priority: P4
  • Status: In Progress
  • Resolution: Unresolved
  • OS: windows_xp
  • CPU: x86
  • Submitted: 2006-10-05
  • Updated: 2023-12-06
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Description
FULL PRODUCT VERSION :
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode)

java version "1.5.0_08"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03)
Java HotSpot(TM) Client VM (build 1.5.0_08-b03, mixed mode, sharing)

java version "1.6.0-beta2"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.6.0-beta2-b72)
Java HotSpot(TM) Client VM (build 1.6.0-beta2-b72, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Windows XP SP2

EXTRA RELEVANT SYSTEM CONFIGURATION :
2GB RAM

A DESCRIPTION OF THE PROBLEM :
Attempting to read a large file in one chunck with FileInputStream.read() with TOO large heap space causes OutOfMemory exception. According to spec, AFIK, even if the read() runs out of space it should just return with as much data as it can, not throw an exception. And in any case there was plenty of memory available, and further more with smaller heap the code runs ok!

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the attached code with heap space (-Xmx) 1300M and 1400M. With 1300M it runs ok, with 1400M it throws an exception. The "bigfile" in my test case had a length of 251 503 002 bytes. I can provide the file, but it should not make difference as the programs just tries to read all the bytes.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Either the FileInputStream.read() should have read the whole file (as there was 1400 MB of memory available and the file size was 'only' 250MB) or it should have read some smaller part of the file and returned the number of bytes read. It should not have thrown an exception.
ACTUAL -
With TOO large heap size the code throws OutOfMemory exception.

ERROR MESSAGES/STACK TRACES THAT OCCUR :

C:\tests>"C:\Program Files\Java\jre1.6.0\bin\java.exe" -Xmx1400m -classpath . r3
d
size: 251503002
buf ok
Exception in thread "main" java.lang.OutOfMemoryError
        at java.io.FileInputStream.readBytes(Native Method)
        at java.io.FileInputStream.read(Unknown Source)
        at r3d.main(r3d.java:19)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.awt.*;
import java.io.*;

public class r3d {

    public static void main(String[] args) {
	try {
		int size = 501*501*501*2;

		FileInputStream fis = new FileInputStream("bigfile"); // Any file with size >= 501*501*501*2

		System.out.println("size: " + size);

		byte buf[] = new byte[size];

		System.out.println("buf ok");


		int bytesRead = fis.read(buf, 0,size);
		System.out.println("Bytes read " + bytesRead);
	}
	catch (Exception e) {
		e.printStackTrace();
		System.out.println(e);
		System.out.println();
	}
    }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Attempting to read the file in multiple smaller chunks seem to work,.

Comments
noreg-other: Affected methods are covered by existing tests.
24-07-2023

A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/14981 Date: 2023-07-21 22:40:00 +0000
21-07-2023

A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk/pull/8235 Date: 2022-04-14 01:40:50 +0000
14-04-2022

The reported OOME was not observed on Linux, macOS, or Windows for the array size 501*501*501*2. Increasing the size to Integer.MAX_VALUE - 8 caused an OOME unless, e.g., -Xmx4G was set on the command line. As [~alanb] mentioned above, a malloc'ed buffer is used in the native code if the size is greater than 8K bytes. As suggested, this could be changed to a smaller value which would limit the number of bytes read(byte[],int,int) is able to read unless looping were added either in the Java or native layer.
18-10-2021

EVALUATION A malloc'ed buffer is used for I/O when byte array is larger than 8k. We should consider limiting the size to something reasonable (say 1MB). Additioning, this temporary buffer could be a thread local rather than malloc'ed buffer to reduce the cost of malloc/free per I/O.
17-08-2009