Currently troubleshooting of deadlocks caused by GCLocker is hard.
The regular diagnostic tools like jstack fail. Jhsdb can produce java stack traces, and sometimes can also produce native stacks. Still, analyzing the failures requires a deep dive into the hotspot source code.
We could change the assert in GCLocker::stall_until_clear() to a guarantee. This will crash the JVM instead of deadlocking, and will produce a stack trace that will point to the area where the deadlock was triggered.
In order to avoid having these failures reported to us, we could change the failure message to something like "A JNI Critical API deadlock has occurred - please run with -Xcheck:jni to diagnose" (credit to [~dholmes] for suggestion)
To reproduce:
create a JNI function, call GetPrimitiveArrayCritical, then repeatedly call NewObject until GC deadlocks.
See JDK-8307185 (and duplicates) for an example of related bug.