The flag LoopMaxUnroll currently behaves a bit unexpected, and does not allow explicit control over unrolling before vectorization and after vectorization.
Such control would be quite helpful to debug performance issues, such as encountered during JDK-8367158, where we want to compare different unrolling factors of vectorized code. For example we may want to compare auto-vectorized code of different super-unrolling factors with fill/copy intrinsics.
--------------------- How LoopMaxUnroll currently works ----------------
The description of the flag is a bit weak:
200 product(intx, LoopMaxUnroll, 16, \
201 "Maximum number of unrolls for main loop") \
202 range(0, max_jint) \
It seems to suggest it might cover the factor of "pre-vec-unroll * super-unroll", i.e. total unroll. But that does not seem to be the case.
In IdealLoopTree::policy_unroll
977 _local_loop_unroll_limit = LoopUnrollLimit;
978 _local_loop_unroll_factor = 4;
979 int future_unroll_cnt = cl->unrolled_count() * 2;
980 if (!cl->is_vectorized_loop()) {
981 if (future_unroll_cnt > LoopMaxUnroll) return false;
982 } else {
983 // obey user constraints on vector mapped loops with additional unrolling applied
984 int unroll_constraint = (cl->slp_max_unroll()) ? cl->slp_max_unroll() : 1;
985 if ((future_unroll_cnt / unroll_constraint) > LoopMaxUnroll) return false;
986 }
So when we are checking if we can unroll, there are these cases:
- scalar loop -> do not unroll more than LoopMaxUnroll
- vector main loop -> do not unroll more than LoopMaxUnroll*slp_max_unroll
- vector drain loop -> do not unroll more than LoopMaxUnroll. Q: can that lead to super-unrolling? Probably not...?
It seems we also always set _local_loop_unroll_factor = 4, which mal be relevant below.
1116 if (phase->C->do_superword()) {
1117 // Only attempt slp analysis when user controls do not prohibit it
1118 if (!range_checks_present() && (LoopMaxUnroll > _local_loop_unroll_factor)) {
1119 // Once policy_slp_analysis succeeds, mark the loop with the
1120 // maximal unroll factor so that we minimize analysis passes
1121 if (future_unroll_cnt >= _local_loop_unroll_factor) {
1122 policy_unroll_slp_analysis(cl, phase, future_unroll_cnt);
1123 }
1124 }
1125 }
So LoopMaxUnroll must be 8 or larger, otherwise we don't do the policy_unroll_slp_analysis. Curious!
-> Investigate!
1127 int slp_max_unroll_factor = cl->slp_max_unroll();
1128 if ((LoopMaxUnroll < slp_max_unroll_factor) && FLAG_IS_DEFAULT(LoopMaxUnroll) && UseSubwordForMaxVector) {
1129 LoopMaxUnroll = slp_max_unroll_factor;
1130 }
We may now update the flag, if still in default mode. But this is a global update... so next time we come through here we cannot update it any more, right? Does this look good at all?
-> Investigation: ok, it does get updated repeatedly. I played around with an example like TestLoopMaxUnrollIncreasing.java
We also use the flag to limit the search for reduction chains in SuperWord:
src/hotspot/share/opto/superword.cpp: PathEnd path_to_phi = find_in_path(n, input, LoopMaxUnroll, has_my_opcode,
src/hotspot/share/opto/superword.cpp: PathEnd path_from_phi = find_in_path(first, input, LoopMaxUnroll, has_my_opcode,
policy_unroll_slp_analysis
- unrolling_analysis sets _local_loop_unroll_factor, mark_passed_slp, and set_slp_max_unroll.
- Maybe we can refactor the code, so we don't set random states all over the place?
- eventually, we then set the _local_loop_unroll_limit, which seems to be the real limit... ah but that is a node limit??? a bit strange... and could possibly lead to inaccurate super-unrolling. The heuristic is also odd.
-> this probably leads to the horrible over-unrolling we have seen when we pass slp analysis but fail to vectorize!
TODO: policy_unroll_slp_analysis - meaning of related fields
TODO: consider deprecating UseSubwordForMaxVector - I see no reason to disable it ever. We also have no tests for its correctness if disabled.
TODO: consider deprecating SuperWordLoopUnrollAnalysis - ah but it is false on arm only... strange!