JDK-8242032 added the following code under lock:

    // Rechecking if the region is coarsened, while holding the lock.
    if (is_region_coarsened(from_hrm_ind)) {
      assert(contains_reference(from), "We just found " PTR_FORMAT " in the Coarse table", p2i(from));
      return;
    }

The problem is the contains_reference() call, that tries to lock the same Mutex while already holding that same lock. Actually the comment above that "if" says that we're holding the lock....

So Mutex code correctly asserts with

  assert(owner() != self) failed: invariant

The fix is to call contains_reference_locked() instead.

The original author of JDK-8242032 already mentioned that he could not reproduce the situation - now we have evidence that there is such a problem. :)