Currently G1 assigns a thread per failed evacuated region. This can in effect serialize the whole process as often (particularly with region pinning) there is only one region to fix up.

Try to improve parallelism when walking over the regions by e.g. recording potential entry points with live object starts when evacuation failure happens for sub-areas of the region during evacuation failure handling.

Note that the BOT should NOT be used as it is not created for young gen regions; using it would recreate it at that point which is basically a full region walk.


Latest implementation scans regions in chunks to bring parallelism, it's based on JDK-8278917 which changes to uses prev bitmap to mark evacuation failure objs.



(I can not upload the attachments.)
Here's the summary of performance data based on latest implementation, basically, it brings better and stable performance than baseline at "Post Evacuate Cleanup 1/remove self forwardee" phase. (Although some regression is spotted when calculate the results in geomean, becuase one pause time from baseline is far too small than others.)

The performance benefit trend is:
 - pause time (Post Evacuate Cleanup 1) is decreased from 76.79% to 2.28% for average time, from 71.61% to 3.04% for geomean, when G1EvacuationFailureALotCSetPercent is changed from 2 to 90 (-XX:ParallelGCThreads=8)
 - pause time (Post Evacuate Cleanup 1) is decreased from 63.84% to 15.16% for average time, from 55.41% to 12.45% for geomean, when G1EvacuationFailureALotCSetPercent is changed from 2 to 90 (-XX:ParallelGCThreads=<default=123>)
( Other common Evacuation Failure configurations are:
-XX:+G1EvacuationFailureALot -XX:G1EvacuationFailureALotInterval=0 -XX:G1EvacuationFailureALotCount=0 )


(unit is ms, please ignore the too precise float number, I guess it should be due to copying from excel)
============= -XX:ParallelGCThreads=8 ==============
-XX:G1EvacuationFailureALotCSetPercent=2
5.785714286	1.342857143	76.79%       // AVG
3.491874057	0.991334526	71.61%       // GEOMEAN
-XX:G1EvacuationFailureALotCSetPercent=5
6.457142857	5.257142857	18.58%       // AVG
3.751329198	4.842453032	-29.09%      // GEOMEAN
-XX:G1EvacuationFailureALotCSetPercent=10
7.7	5.857142857	23.93%                       // AVG
6.438810941	4.01726936	37.61%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=20
24.58571429	19.61428571	20.22%       // AVG
23.91174748	19.30619931	19.26%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=30
38.94285714	36.62857143	5.94%       // AVG
37.68215788	34.95941805	7.23%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=50
61.64285714	58.01428571	5.89%       // AVG
58.48164395	54.11292376	7.47%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=70
82.91428571	79.92857143	3.60%       // AVG
77.74442286	74.19262861	4.57%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=90
96.02857143	93.84285714	2.28%       // AVG
89.42483883	86.70240129	3.04%       // GEOMEAN



============= -XX:ParallelGCThreads=<default=123> ==============
-XX:G1EvacuationFailureALotCSetPercent=2
5.728571429	2.071428571	63.84%       // AVG
4.55615823	2.031440622	55.41%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=5
9.371428571	3.4	63.72%                       // AVG
8.921695255	3.189298958	64.25%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=10
10.37142857	4.842857143	53.31%       // AVG
9.36881833	4.583811393	51.07%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=20
15.18571429	10.72857143	29.35%       // AVG
14.81067789	10.2772486	30.61%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=30
21.01428571	18.74285714	10.81%       // AVG
20.70456459	18.26632857	11.78%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=50
34.58571429	30.07142857	13.05%       // AVG
33.57743078	28.85928625	14.05%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=70
47.17142857	40.34285714	14.48%       // AVG
45.28903319	39.17734325	13.49%       // GEOMEAN

-XX:G1EvacuationFailureALotCSetPercent=90
58.71428571	49.81428571	15.16%       // AVG
54.65737668	47.85414364	12.45%       // GEOMEAN