FULL PRODUCT VERSION :
Linux slc00vfp 2.6.18-238.0.0.0.1.el5xen #1 SMP Tue Jan 4 09:38:01 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
Although this affects a variety of JDKs/JREs, including 1.5 through 1.7 and openjdk6/7.
ADDITIONAL OS VERSION INFORMATION :
Linux slc00vfp 2.6.18-238.0.0.0.1.el5xen #1 SMP Tue Jan 4 09:38:01 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
But this issue is generic to all OSes, since the limitation is in sun.nio.ch.FileChannelImpl.java.
A DESCRIPTION OF THE PROBLEM :
The code in FileChannelImpl.map() lets you create a single mmap of any size up to but not including 2.0 GiB (2^31 bytes). The code that limits this is:
if (size > Integer.MAX_VALUE)
throw new IllegalArgumentException("Size exceeds Integer.MAX_VALUE");
Presumably this is because the MappedByteBuffer, which extends ByteBuffer, has get/put methods that take an integer index into the buffer. However the range of that index includes zero, so in theory one could have a buffer with exactly (Integer.MAX_VALUE + 1) values that are indexed from 0..Integer.MAX_VALUE.
This looks like an off-by-one Bug: there's no physical or logical limitation that needs to enforce the maximum size of a memory map has to be less than 2.0 GiB. It also doesn't make logical sense that you can map ((1 << 31) - 1) bytes but you cannot map exactly (1 << 31) bytes. In fact, there could be potential performance implications if you don't map a number of bytes which is a multiple of the OS's page size.
In my use case, I want to map a 4.0 GiB file using multiple mmap'd objects. If FileChannel.map() allowed a 2.0 GiB map I would need two consecutive (with respect to the file, not in memory) maps and I could use integer bit manipulation to decide which of the MappedByteBuffers to use. But because of this off-by-one bug in FileChannelImpl, I have a couple of options:
1). create two ((1 << 31) - 1)-byte maps and a third 2-byte map, and use integer division and modulus; doing this will impact performance.
2). create three more-equally-sized maps, again using integer division and modules; this also impacts performance.
3). create four (1 << 30)-byte maps, and use bit shifting and masking to obtain the offset into the ByteBuffer and to choose the desired ByteBuffer.
It would be better if the FileChannel implementation allowed exactly (1 << 31) bytes instead of ((1 << 31) - 1) bytes in a single mmap. My suggestion is to change the code (and the corresponding Javadoc) to something like:
if (size > Integer.MAX_VALUE + 1)
throw new IllegalArgumentException("Size exceeds maximum (Integer.MAX_VALUE + 1)");
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Try to allocate exactly a 2.0 GiB buffer:
channel.map(MapMode.READ_ONLY, 0, (1L << 31)); // this throws an exception
channel.map(MapMode.READ_ONLY, 0, (1L << 31) - 1); // this does not
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Ideally, this code should succeed:
channel.map(MapMode.READ_ONLY, 0, (1L << 31));
but this code should fail:
channel.map(MapMode.READ_ONLY, 0, (1L << 31) + 1)
ACTUAL -
IllegalArgumentException("Size exceeds Integer.MAX_VALUE")
ERROR MESSAGES/STACK TRACES THAT OCCUR :
IllegalArgumentException("Size exceeds Integer.MAX_VALUE")
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
class FileChannelMapIsOffByOne
{
public static void main(String[] args) throws Throwable
{
RandomAccessFile raf = new RandomAccessFile("/dev/zero", "r");
FileChannel fc = raf.getChannel();
MappedByteBuffer mbb = fc.map(MapMode.READ_ONLY, 0, (1 << 31));
System.out.println("never gets here");
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
Take FileChannelImpl.java out of the JDK's src.zip, change:
if (size > Integer.MAX_VALUE)
inside map() to:
if (size > Integer.MAX_VALUE + 1)
Compile this class, throw it in the JDK's rt.jar or specify the class using endorsed dirs.