Bug ID: JDK-8317440 Lock rank checking fails when code root set is modified with the Servicelock held after JDK-8315503

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 21	JDK 22
21.0.4-oracleFixed	22 b19Fixed

I ran tier7 testing with JDK 22 and hit this issue on Windows-x64:

#  Internal Error (\\workspace\\open\\src\\hotspot\\share\\runtime\\mutex.cpp:448), pid=7148, tid=38704
#  assert(false) failed: Attempting to acquire lock ConcurrentHashTableResize_lock/nosafepoint-3 out of order with lock Service_lock/service -- possible deadlock
#
# JRE version: Java(TM) SE Runtime Environment (22.0) (fastdebug build 22-internal-2023-10-02-1945520.vladimir.kozlov.jdkgit)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 22-internal-2023-10-02-1945520.vladimir.kozlov.jdkgit, compiled mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64)

It could be caused by JDK-8315503.

A pull request was submitted for review. URL: https://git.openjdk.org/jdk21u-dev/pull/476 Date: 2024-04-08 20:00:14 +0000
10-04-2024
Fix request [21u] I backport this for parity with 21.0.4-oracle. Required follow-up of JDK-8315503 and included in that backport. Clean backport.
08-04-2024
[~tschatzl] Is there a command line with options that can reproduce this problem?
16-01-2024
The fix for this bug is integrated in jdk-22+19-1444.
09-10-2023
Changeset: 0cf1a558 Author: Thomas Schatzl <tschatzl@openjdk.org> Date: 2023-10-09 08:27:56 +0000 URL: https://git.openjdk.org/jdk/commit/0cf1a558bacf18d9fc41e43fb5e9eba39dc51f2e
09-10-2023
A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/16062 Date: 2023-10-05 17:19:35 +0000
06-10-2023
The reason is that registering an nmethod can take the CHT specific lock; this one is fixed to ConcurrentHashTableResize_lock/nosafepoint-3 which is larger (in numeric value) than Service_lock/service. JDK-8315503 changed the priorities of the CHT lock already due to similar problems with CodeCache_lock, but that proved insuffficient. There is no deadlock in product because the CHTs G1 uses are not locking anything themselves in any of the CHT supplied code. Edit: they can do that because the global synchronization can lock the ThreadSMR_lock. So the previous fix to just make the CHT locks a very low rank does not work (without also changing ThreadSMR_lock).
04-10-2023
Both test failed when run with -Xcomp: KitchenSink failed with additional flags: -Xcomp -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:+TieredCompilation -XX:+VerifyOops RunThese30M failed with additional flags: -Xcomp -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:-TieredCompilation -XX:-DoEscapeAnalysis
03-10-2023

Relates :	JDK-8323685 - PrintSystemDictionaryAtExit has mutex rank assert
Relates :	JDK-8315503 - G1: Code root scan causes long GC pauses due to imbalanced iteration
Relates :	JDK-8326092 - Pause Remark sometimes has extremely long pause times on class unloading
Relates :	JDK-8318109 - Writing JFR records while a CHT has taken its lock asserts in rank checking