Failure to allocate a JNI handle (due to malloc failure) is presently reported using vm_exit_out_of_memory (when creating a new JNIHandleBlock in JNIHandles::allocate_block). That's not the behavior described in the JNI specification, which says that for out of memory failures
- NewGlobalRef will return NULL.
- NewLocalRef may return NULL.
- NewWeakGlobalRef will return NULL and throw OutOfMemoryError.
- PushLocalFrame will return a negative number and a pending OutOfMemoryError.
Note that OutOfMemoryError doesn't really seem like the right exception for running out of system memory, but that's what the JNI spec says.