In debug.cpp report_vm_out_memory is defined as follows:

void report_vm_out_of_memory(const char* file_name, int line_no, size_t size, const char* message) {
  if (Debugging || assert_is_suppressed(file_name, line_no))  return;

  // We try to gather additional information for the first out of memory
  // error only; gathering additional data might cause an allocation and a
  // recursive out_of_memory condition.

  const jint exiting = 1;
  // If we succeed in changing the value, we're the first one in.
  bool first_time_here = Atomic::xchg(exiting, &_exiting_out_of_mem) != exiting;

  if (first_time_here) {
    Thread* thread = ThreadLocalStorage::get_thread_slow();
    VMError(thread, size, message, file_name, line_no).report_and_die();
  }

  // Dump core and abort
  vm_abort(true);
}

The detection of recursive calls at this level is, as far as I can see, unnecessary, because a second call to report_and_die() will itself detect there is an existing error report in progress and block the thread. Further, the above logic is undesirable because if a second OOM condition is encountered in another thread then that thread will cause an immediate abort and core dump, while the first thread is trying to report details of the first error.