While looking at the review for JDK-8131734 there has been a discussion whether at heap shrinking time the remembered sets should be cleared. Apparently this is not the case even during full GC after visual inspection of the code.
While the remembered set will be reclaimed anyway when reallocating that region, it might take a while until a region is reallocated. So somewhat more eagerly clearing the remembered set could help average memory usage.
Check whether this is actually true (remembered sets will be cleared only very lazily), and if so, more eagerly clean the remembered set at that time.