See email from Chris Dennis <###@###.###>:
A signed intx type is used for the MaxDirectMemorySize which limits its value to under 2G. But even on a 32-bit system a value >2GB is not out of the question.
Simply changing to an unsigned unitx type fixes the problem but requires a change in the way we specify "use the default". On the VM size a value of zero now means "use the default" but that causes -1 to be passed on the system property as expected by the JDK side of things.