Particularly on larger heap the "GC aggregate-data" phase of remark uses up a significant amount of time.

E.g. take a look at the time spent here in G1 remark with +PrintReferenceGG:

584.680: #389: [GC remark
  584.680: #389: [Finalize Marking, 0.0110132 secs]
    584.691: #389: [GC ref-proc
      584.691: #389: [SoftReference, 0 refs, 0.0002637 secs]
      584.691: #389: [WeakReference, 206 refs, 0.0003279 secs]
      584.691: #389: [FinalReference, 26 refs, 0.0015083 secs]
      584.693: #389: [PhantomReference, 1 refs, 0.0003140 secs]
      584.693: #389: [JNI Weak Reference, 0.0000502 secs]
    , 0.0025383 secs]
  584.693: #389: [Unloading, 0.0630265 secs]
  584.756: #389: [GC aggregate-data, 0.4297980 secs]
, 0.5155041 secs] 

Accounting for approximately 80% of the (long) pause.

Inspecting the code of the GC aggregate-data method, it looks as if this work can be moved to a concurrent phase between Remark and Cleanup.

Apart from a single addition to a global variable, everything is local.