The ergonomically determined maximum number of fine PRTs for a single remembered set is determined by G1RSetRegionEntriesBase.

If a remembered set has more than that fine PRTs, it starts coarsening random others.

Ergonomics only takes region size into account, but not actual number of regions, so if you have a large heap (and/or small region size), the default ergonomics are too low.

G1RSetRegionEntries should be sized with  the number of regions too to avoid the significant performance penalty.