JDK-7147951 : FileChannel.map() cannot map exactly 2 GiB
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.nio
  • Affected Version: 6u29
  • Priority: P4
  • Status: Closed
  • Resolution: Not an Issue
  • OS: linux_redhat_5.0
  • CPU: x86
  • Submitted: 2012-02-22
  • Updated: 2012-03-20
  • Resolved: 2012-02-23
Related Reports
Relates :  
Description
FULL PRODUCT VERSION :
Linux slc00vfp 2.6.18-238.0.0.0.1.el5xen #1 SMP Tue Jan 4 09:38:01 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

Although this affects a variety of JDKs/JREs, including 1.5 through 1.7 and openjdk6/7.

ADDITIONAL OS VERSION INFORMATION :
Linux slc00vfp 2.6.18-238.0.0.0.1.el5xen #1 SMP Tue Jan 4 09:38:01 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

But this issue is generic to all OSes, since the limitation is in sun.nio.ch.FileChannelImpl.java.

A DESCRIPTION OF THE PROBLEM :
The code in FileChannelImpl.map() lets you create a single mmap of any size up to but not including 2.0 GiB (2^31 bytes).  The code that limits this is:

        if (size > Integer.MAX_VALUE)
            throw new IllegalArgumentException("Size exceeds Integer.MAX_VALUE");

Presumably this is because the MappedByteBuffer, which extends ByteBuffer, has get/put methods that take an integer index into the buffer.  However the range of that index includes zero, so in theory one could have a buffer with exactly (Integer.MAX_VALUE + 1) values that are indexed from 0..Integer.MAX_VALUE.

This looks like an off-by-one Bug: there's no physical or logical limitation that needs to enforce the maximum size of a memory map has to be less than 2.0 GiB.  It also doesn't make logical sense that you can map ((1 << 31) - 1) bytes but you cannot map exactly (1 << 31) bytes.  In fact, there could be potential performance implications if you don't map a number of bytes which is a multiple of the OS's page size.

In my use case, I want to map a 4.0 GiB file using multiple mmap'd objects.  If FileChannel.map() allowed a 2.0 GiB map I would need two consecutive (with respect to the file, not in memory) maps and I could use integer bit manipulation to decide which of the MappedByteBuffers to use.  But because of this off-by-one bug in FileChannelImpl, I have a couple of options:
1). create two ((1 << 31) - 1)-byte maps and a third 2-byte map, and use integer division and modulus; doing this will impact performance.
2). create three more-equally-sized maps, again using integer division and modules; this also impacts performance.
3). create four (1 << 30)-byte maps, and use bit shifting and masking to obtain the offset into the ByteBuffer and to choose the desired ByteBuffer.

It would be better if the FileChannel implementation allowed exactly (1 << 31) bytes instead of ((1 << 31) - 1) bytes in a single mmap.  My suggestion is to change the code (and the corresponding Javadoc) to something like:

        if (size > Integer.MAX_VALUE + 1)
            throw new IllegalArgumentException("Size exceeds maximum (Integer.MAX_VALUE + 1)");

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Try to allocate exactly a 2.0 GiB buffer:

  channel.map(MapMode.READ_ONLY, 0, (1L << 31)); // this throws an exception
  channel.map(MapMode.READ_ONLY, 0, (1L << 31) - 1); // this does not

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Ideally, this code should succeed:

  channel.map(MapMode.READ_ONLY, 0, (1L << 31));

but this code should fail:

  channel.map(MapMode.READ_ONLY, 0, (1L << 31) + 1)
ACTUAL -
IllegalArgumentException("Size exceeds Integer.MAX_VALUE")

ERROR MESSAGES/STACK TRACES THAT OCCUR :
IllegalArgumentException("Size exceeds Integer.MAX_VALUE")

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
class FileChannelMapIsOffByOne
{
    public static void main(String[] args) throws Throwable
    {
        RandomAccessFile raf = new RandomAccessFile("/dev/zero", "r");
        FileChannel fc = raf.getChannel();
        MappedByteBuffer mbb = fc.map(MapMode.READ_ONLY, 0, (1 << 31));
        System.out.println("never gets here");
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Take FileChannelImpl.java out of the JDK's src.zip, change:
   if (size > Integer.MAX_VALUE)
inside map() to:
   if (size > Integer.MAX_VALUE + 1)
Compile this class, throw it in the JDK's rt.jar or specify the class using endorsed dirs.

Comments
EVALUATION Note that submitter wants to map 2^31 bytes but that Integer.MAX_VALUE is 2^31-1. Inability to map 2^31 bytes is a limitation of the spec; the doc for FileChannel.map() says that size must not be greater than Integer.MAX_VALUE. Fixing this would be a spec change (not a simple code change as suggested), plus changes to a bunch of other APIs. For example, Buffer.capacity() would also have to be changed from int to long. This is covered by 6347833.
23-02-2012

EVALUATION FileChannel.map appears to be working correctly. If a size of Integer.MAX_VALUE is specified then it maps 0 to Integer.MAX_VALUE-1 as expected.
23-02-2012