C2 has max unroll limit hard coded:
#define MAX_UNROLL 16 // maximum number of unrolls for main loop
It limits size of vectors (byte vectors) which are used in AVX instructions. There is no feed back from vectorization code to loop unrolling because superword optimization is executed at the end of all loop opts. We can make policy_unroll() method more intelligent to recognize vectorization cases but it is a lot more code changes. For now just use new LoopMaxUnroll flag instead of MAX_UNROLL constant.