JDK-5092131 : using 1 MB pagesize throws "not enough space" error with 32bit JVM
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.nio
  • Affected Version: 1.4.2_05
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • OS: generic
  • CPU: generic
  • Submitted: 2004-08-25
  • Updated: 2006-12-06
  • Resolved: 2006-12-06
Related Reports
Duplicate :  
JDK-6417205 - (fc) Out of address space because mapped ByteBuffers are not freed
Relates :  
JDK-4797189 - Finalizers not called promptly enough
Relates :  
JDK-6751792 - OutOfMemory error from native leak in PNGEnoder class?
Relates :  
JDK-5072161 - OutOfMemoryError while using GZIPOutputStream
Description
We have a problem when we are accessing a huge file (~6GB) using NIO and
MappedByteBuffers. Since file is huge, I tried to map only portions of
the file  - no more than 16 portions 1MB each. And eventually I got an
IO exception "Can not allocate memory". I was not able to find any leaks
or other problems in my program. It works fine with relatively small
files. Then I searched news groups and found posting about similar
issue. So, I ran a simple test program on the same file and reproduced
the problem - once I set the portion size to 1MB, I am getting an exception.

Below is an output from successfull run (when portion is 100KB) and failing run (with 1MB). You can also see the version of JVM I am running. I am able to reproduce this on Linux and on Windows 2000.

polly:/tmp 3> java -version
java version "1.4.2_03"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_03-b02)
Java HotSpot(TM) Client VM (build 1.4.2_03-b02, mixed mode)
polly:/tmp 4> java -cp . Test $RDB/adt040803.dat 10000 100000
Mapping /usr/local/instinet/RTS/var/db/2004_08_03/adt040803.dat
  size of file: 6337593344
  size of page: 100000
  iterations  : 10000
polly:/tmp 5> java -cp . Test $RDB/adt040803.dat 10000 1000000 | less 2>&1
Mapping /usr/local/instinet/RTS/var/db/2004_08_03/adt040803.dat
  size of file: 6337593344
  size of page: 1000000
  iterations  : 10000
Exception occurred at count 1927 trying to map from 3557000000 to 3558000000
(1927 faults so far)
Exception in thread "main" java.io.IOException: Cannot allocate memory
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:705)
        at Test.main(Test.java:53)
polly:/tmp 6>
Comments
EVALUATION -- 6417205 provides some relief for this issue.
05-12-2006
WORK AROUND Use 64bit JVM along with 64bit OS. --------------------------------------------------------------- When allocating large MappedByteBuffers, explicitly call System.gc(); System.runFinalization(); once in a while (say, every 100MB of mapped memory). (Of course, this should be done only as a last resort, and there is no guarantee that the mapped memory will indeed be freed by this) ###@###.### 2004-08-27
27-08-2004
EVALUATION Here is a simpler test that, I believe clarifies the problem: --------------------------------------------------------------------- import java.io.*; import java.nio.*; import java.nio.channels.*; class Test { public static void main(String[] args) throws Exception { if (args.length != 3) { System.err.println("Usage: java Test FILENAME COUNT MAPSIZE"); System.exit(1); } File f = new File(args[0]); int count = Integer.parseInt(args[1]); int mapsize = Integer.parseInt(args[2]); FileChannel fc = new RandomAccessFile(f, "r").getChannel(); long size = fc.size(); long start = 0; for (int i = 0; i < count; i++) { //if (i % 1000 == 0) { System.gc(); System.runFinalization();} try { long offset = (long)(Math.random() * size); fc.map(FileChannel.MapMode.READ_ONLY, Math.min(size - mapsize, offset), mapsize); } catch (IOException e) { System.err.println("Failed when count="+i); throw e; } } } } --------------------------------------------------------------------- I think we're seeing the usual problem with Java's inability to collect non-heap-memory resources in a timely fashion. The finalizable objects which can unmap the memory will be collected, but the GC is not aware of the urgency, since they appear to be simple small Java objects. Most Java objects consuming non-heap resources have a close() method, but not MappedByteBuffer. The scarce resource here is not physical or virtual memory, but *address space*. The submitter should confirm the analysis by trying the above program with and without the commented out line or by inserting such a line into the application code. ###@###.### 2004-08-27
27-08-2004