Right now, we have a flag "OnOutOfMemoryError=..." You can do a lot of things with it, for example -XX:OnOutOfMemoryError='kill -9 %p' to generate a core file on OOM.
However, exiting and generating core files are two common cases that you may want to use when debugging. Instead of relying on command line tools (which is also harder on Windows), I suggest we implement two special cases. These were present in JRockit, so I suggest we use the same syntax (see http://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm )
There two options in JRockit were:
When you enable this option, the JVM exits on the first occurrence of an out-of-memory error. It can be used if you prefer restarting an instance of the JVM rather than handling out of memory errors.
If this option is enabled, when an out-of-memory error occurs, the JVM crashes and produces text and binary crash files.