FULL PRODUCT VERSION :
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)


FULL OS VERSION :
Linux 3.11.0-24-generic #42-Ubuntu SMP Fri Jul 4 21:19:31 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
Setting -Xmx incorrectly limits the total amount of direct memory you can allocate using ByteBuffer.allocateDirect.

java.nio.Bits.reserveMemory throws an OutOfMemoryError when when the request will cause the totalCapacity to go above "Bits.maxMemory".

Bits.maxMemory is set using sun.misc.VM.maxDirectMemory().

VM.maxDirectMemory() returns the static VM.directMemory.

When the property sun.nio.MaxDirectMemorySize is left unset (when -XX:MaxDirectMemorySize=<size> option isn't used), VM.directMemory is set to the response from Runtime.getRuntime().maxMemory().

Runtime.getRuntime().maxMemory() is a native call that results in a call to Universe::heap()->max_capacity.

Universe::heap() IS THE JAVA HEAP! and max_capacity() returns a number slightly smaller than whatever is set as -Xmx (this can be verified by simply calling Rumtime.getRunteim().maxMemory() from java.

This is completely wrong and against all documentation (e.g. see the documentation for XX:MaxDirectMemorySize especially with reference to the default) or common sense use of the DirectByteBuffer to use OFF-HEAP memory.

This was also a problem at least in 7u40.


THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Did not try

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
call ByteBuffer.allocateDirect() with a capacity that will fit comfortably on the machine.

Limit the java heap using -Xmx to something smaller than that capacity.

The allocation will fail with an OutOfMemoryError.

If you then set the XX:MaxDirectMemorySize option it will work.


EXPECTED VERSUS ACTUAL BEHAVIOR :
The call to ByteBuffer.allocateDirect using a capacity that will comfortably fit on the machine, without setting the -XX:MaxDirectMemorySize, should NOT result in an OutOfMemoryError just because -Xmx is set. As the documentation for -XX:MaxDirectMemorySize says "The default value is zero, which means the maximum direct memory is unbounded."
ERROR MESSAGES/STACK TRACES THAT OCCUR :
java.lang.OufOfMemoryError: Direct buffer memory

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------

  // Set the max heap space to 10m using - Xmx10m
  // Run on a machine with plenty of memory
  System.out.println(Runtime.getRuntime().maxMemory());
  ByteBuffer.allocateDirect(1024 * 1024 * 11);


---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :

Supply a really high value using -XX:MaxDirectMemorySize. E.g. -XX:MaxDirectMemorySize=999g