I stumbled upon this when doing some Shenandoah work. The development code tried to lock the "leaf" lock, while already holding the "access" lock. Normally it would have been detected by VM, but instead, we tried to recursively acquire tty_lock for Thread::print_owned_locks. But that tty_lock is still a higher ranked lock, so deadlock detection triggers over and over again until we run out of stack and crash hard.

Add this to test_mutex_rank.cpp:

TEST_VM_ASSERT_MSG(MutexRank, mutex_wait_access_leaf,
                   ".* Attempting to acquire lock mutex_rank_leaf/11 out of order with lock mutex_rank_access/1 "
                   "-- possible deadlock") {
  JavaThread* THREAD = JavaThread::current();
  ThreadInVMfromNative invm(THREAD);

  Mutex* mutex_rank_access = new Mutex(Mutex::access, "mutex_rank_access", false, Mutex::_safepoint_check_never);
  Mutex* mutex_rank_leaf = new Mutex(Mutex::leaf, "mutex_rank_leaf", false, Mutex::_safepoint_check_never);

  mutex_rank_access->lock_without_safepoint_check();
  mutex_rank_leaf->lock_without_safepoint_check();
  mutex_rank_leaf->unlock();
  mutex_rank_access->unlock();
}

Then observe the hard crash (no asserts, no hs_err_*), and see this in GDB:

...
#3455 outputStream::print_cr at ostream.cpp:158
#3456 0x00007f596a7d445b in Thread::print_owned_locks_on at thread.cpp:836
#3457 0x00007f596a2d0fc2 in Thread::print_owned_locks at thread.hpp:768
#3458 Mutex::check_rank at mutex.cpp:415
#3459 0x00007f596a2d1805 in Mutex::lock_without_safepoint_check at mutex.cpp:131
#3460 0x00007f596a38da64 in defaultStream::hold at ostream.cpp:845
#3461 0x00007f596a38dc1e in defaultStream::write at ostream.cpp:873
#3462 0x00007f596a38a590 in outputStream::do_vsnprintf_and_write_with_automatic_buffer at ostream.cpp:131
#3463 0x00007f596a38b2ef in outputStream::do_vsnprintf_and_write at ostream.cpp:144
#3464 outputStream::print_cr at ostream.cpp:158
#3465 0x00007f596a7d445b in Thread::print_owned_locks_on at thread.cpp:836
#3466 0x00007f596a2d0fc2 in Thread::print_owned_locks  at thread.hpp:768
#3467 Mutex::check_rank at mutex.cpp:415
#3468 0x00007f596a2d1805 in Mutex::lock_without_safepoint_check at mutex.cpp:131
#3469 0x00007f5968fdde5c in test_MutexRank_mutex_wait_access_leaf_ at test_mutex_rank.cpp:117
#3470 child_MutexRank_mutex_wait_access_leaf_ at test_mutex_rank.cpp:107

Workarounds:
 - We could have ranked "tty" lower than "access" lock, but that effectively reverts JDK-8214315.
 - We could add Mutex::tty to check_can_be_skipped, but that effectively disables the tty deadlock checking?