Currently, after JDK-8202049, G1 just bails out if there is not enough mark stack during reference processing in the Remark pause.

A comment in the CR also states that:

"We decided that we will change the marking to simply expand the global mark stack when it goes out of memory, until MarkStackSizeMax. We will exit the VM if that is not enough global mark stack memory available for processing. We will remove the re-scan of the region as it is already complicated enough and adding the condition that we rescan only outside of weak reference processing makes it even worse. "

This has not implemented until now as JDK-8280029 indicates, so implement global mark stack resizing (until MarkStackSizeMax) in this case *without* rescanning the heap (which would cause errors in reference processing).