Context: we are debugging an OutOfMemoryError from filling up NIO direct memory (-XX:MaxDirectMemorySize). We tried to use -XX:+HeapDumpOnOutOfMemoryError to collect heap dumps, and found out the flag does not work for direct memory OutOfMemoryError.

There are a few JVM flags related to handling OutOfMemoryError:
-XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError
-XX:+CrashOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError

The work for handling these flags is in `report_java_out_of_memory(const char*)` in debug.cpp. However, `report_java_out_of_memory()` is only called from HotSpot's C++ code, not when a JDK Java class throws OutOfMemoryError directly.

The case relevant to our direct memory investigation is in java/nio/Bits.java line 175 in `reserveMemory()`. Currently there are 24 occurrences of "throw new OutOfMemoryError" in JDK classes. None of them respects the 4 OutOfMemoryError-handling JVM flags.

I propose we add a JNI function that calls `report_java_out_of_memory()`, possibly in jdk/internal/misc/VM.java. Then add a call to this function before the 24 places of throwing OutOfMemoryError.