We have seen one of the cases in our production, where we have TTSP hiccups. Profiling shows the hiccup is in monitor deflation thread, which is busy unlinking the monitors. It is batched with `MonitorDeflationMax` (default: 1M) and checks safepoint after each batch.

However, the current default for the batch is unfortunately very high. If we spend just a 1ns per unlinking monitor, 1M batch size means we would not respond to safepoint request for about 1ms. In practice, it seems the delays are up to 30 ms. This gives a major latency hiccup.

Here is the simplest reproducer I have:

```
import java.util.ArrayList;
import java.util.List;

public class MonitorChurn {
   static final int THREAD_COUNT = Integer.getInteger("threads", 10);
   static final int MONITORS_PER_THREAD = Integer.getInteger("monitors", 1_000_000);

   static class HolderThread extends Thread {
     Object[] objs = new Object[MONITORS_PER_THREAD];

     public void run() {
       try {
         for (int c = 0; c < objs.length; c++) {
            objs[c] = new Object();
         }
         while (true) {
           for (int c = 0; c < objs.length; c++) {
             synchronized (objs[c]) {}
           }
         }
       } catch (Exception e) {
         e.printStackTrace();
       }
     }
   }

   static Object sink;

   public static void main(String... args) throws Exception {
     for (int c = 0; c < THREAD_COUNT; c++) {
       Thread t = new HolderThread();
       t.start();
     }
     while (true) {
       sink = new byte[1000000];
       Thread.sleep(1);
     }
   }
}
```

I added two logging statements in deflation/unlinking code to see when we are actually hitting the deflation/unlinking paths. 

```
diff --git a/src/hotspot/share/runtime/synchronizer.cpp b/src/hotspot/share/runtime/synchronizer.cpp
index c83df922e0d..a5a16c56968 100644
--- a/src/hotspot/share/runtime/synchronizer.cpp
+++ b/src/hotspot/share/runtime/synchronizer.cpp
@@ -132,6 +132,7 @@ size_t MonitorList::unlink_deflated(Thread* current, LogStream* ls,
   ObjectMonitor* prev = nullptr;
   ObjectMonitor* head = Atomic::load_acquire(&_head);
   ObjectMonitor* m = head;
+  log_warning(monitorinflation)("IN UNLINK PATH: " SIZE_FORMAT " MONITORS TO UNLINK", count());
   // The in-use list head can be null during the final audit.
   while (m != nullptr) {
     if (m->is_being_async_deflated()) {
@@ -1688,6 +1689,8 @@ size_t ObjectSynchronizer::deflate_idle_monitors(ObjectMonitorsHashtable* table)
   size_t unlinked_count = 0;
   size_t deleted_count = 0;
   if (deflated_count > 0 || is_final_audit()) {
+    log_warning(monitorinflation)("IN DEFLATION PATH: " SIZE_FORMAT " MONITORS DEFLATED", deflated_count);
+
     // There are ObjectMonitors that have been deflated or this is the
     // final audit and all the remaining ObjectMonitors have been
     // deflated, BUT the MonitorDeflationThread blocked for the final

```

The result is like this:

```
$ /java -XX:+UnlockDiagnosticVMOptions -Xmx256m -XX:GuaranteedAsyncDeflationInterval=1000 -XX:+UseParallelGC -XX:LockingMode=0 -XX:MonitorDeflationMax=1000000 -Xlog:safepoint+stats MonitorChurn.java

[10.589s][info   ][safepoint,stats ] VM Operation                 [ threads: total initial_running ][ time:       sync    cleanup       vmop      total ] page_trap_count
[10.589s][info   ][safepoint,stats ] ParallelGCFailedAllocation   [             20              10 ][            62375       2541   62300334   62365250 ]               0
[10.705s][info   ][safepoint,stats ] ParallelGCFailedAllocation   [             20              10 ][            42875       1375   62180458   62224708 ]               0
[10.756s][warning][monitorinflation] IN DEFLATION PATH: 1000000 MONITORS DEFLATED
[10.757s][warning][monitorinflation] IN UNLINK PATH: 2399912 MONITORS TO UNLINK
[10.834s][info   ][safepoint,stats ] ParallelGCFailedAllocation   [             20              11 ][         12407500       7208   63076667   75491375 ]               1
[10.953s][info   ][safepoint,stats ] ParallelGCFailedAllocation   [             20              11 ][            59208       2042   65406500   65467750 ]               0
[11.067s][info   ][safepoint,stats ] ParallelGCFailedAllocation   [             20              10 ][            26292       1917   61031375   61059584 ]               0
[11.182s][info   ][safepoint,stats ] ParallelGCFailedAllocation   [             20              10 ][            40334       1708   62094458   62136500 ]               0
[11.294s][info   ][safepoint,stats ] ParallelGCFailedAllocation   [             20              10 ][            43083       2167   58604166   58649416 ]               0
[11.404s][info   ][safepoint,stats ] ParallelGCFailedAllocation   [             20              10 ][            43083       2542   56941542   56987167 ]               0
[
```

Near the 2.4M monitor unlinking path we have incurred 12.4 ms TTSP latency.

Naively, one can tune down `MonitorDeflationMax`, but it has impact on the number of monitors we actually deflate per one monitor deflation phase, which unfortunately creeps up monitor population, as deflater thread is less active. 

We need to take a deeper look into safepoint checking policy during unlinkage. It is interesting that deflation loop actually checks for pending safepoint on every monitor. It is not immediately clear why unliking requires batching. If it does require batching for good reason, we might want to split the thresholds into `MonitorDeflationMax` (which would continue to drive deflation batch) and `MonitorUnlinkMax` (which would drive the batching for unlinkage).