Bug ID: JDK-8031751 When doing multiple identical runs in the same environment, benchmark peak scores vary widely for octane benchmarks on Nashorn

Type: Bug
Component: hotspot
Sub-Component: compiler
Affected Version: 9

Priority: P3
Status: Resolved
Resolution: Cannot Reproduce

Submitted: 2014-01-15
Updated: 2016-03-24
Resolved: 2016-03-24

JDK 9
9Resolved

Rerunning e.g. octane.richards, the same number of runs on the same machine with the same binary gives me everything between 53000 ops/s and 123000 ops/s on my machine at steady state. No more code is generated at this steady state, yet we have a huge diversion between peak scores between runs.

I understand Roland already has a partial fix for this. Filing a bug to get it tracked.

Roland- can you elaborate a little bit what the reason for this variation is in this CR, please?

We'd be willing to take your partial fix that still has some sway left into JDK9 and file another CR for any additional work that might come later.

There were some bugs fixed in method handle inlining. I tried latest jdk 9 build and don't see much variation in Richards peak scores: Result: 330.567 ��(99.9%) 0.484 us/op [Average] Statistics: (min, avg, max) = (322.721, 330.567, 348.021), stdev = 3.268 Confidence interval (99.9%): [330.083, 331.050] Benchmark Mode Samples Score Score error Units o.m.b.s.o.RichardsBench.test avgt 500 330.567 0.484 us/op
24-03-2016
With [~roland] gone, [~vlivanov] could you have a look?
18-03-2016
Marcus reports he still sees performance variation with richards. I can reproduce them. On my laptop richards score peaks anywhere from slightly below 140000 to around 190000. When running with JFR and looking at the data collected, the hot methods that are reported look very different between a fast run and a slow run. For instance, fast run: Class Sample Count Percentage(%) jdk.nashorn.internal.scripts.Script$Recompilation$156$richards.release(Object, int) 939 43.392 jdk.nashorn.internal.scripts.Script$richards.addTo(Object, Object) 286 13.216 java.lang.invoke.LambdaForm$DMH.1190900417.invokeStatic_LL_L(Object, Object, Object) 283 13.078 jdk.nashorn.internal.scripts.Script$Recompilation$123$richards.schedule(Object) 158 7.301 slow run: Stack Trace Sample Count Percentage(%) java.lang.invoke.LambdaForm$DMH.1190900417.invokeStatic_LL_L(Object, Object, Object) 405 22.425 java.lang.Integer.valueOf(int) 287 15.891 java.lang.invoke.LambdaForm$MH.107241811.linkToCallSite(Object, Object) 204 11.296 java.lang.invoke.LambdaForm$MH.517693873.guard(Object, Object, Object, Object) 131 7.254 And if I look at why more time is spent in LambdaForm$DMH.1190900417.invokeStatic_LL_L in the slow run I see: Stack Trace Sample Count Percentage(%) java.lang.invoke.LambdaForm$DMH.1190900417.invokeStatic_LL_L(Object, Object, Object) 405 22.425 java.lang.invoke.LambdaForm$MH.514453513.guardWithCatch(Object, Object, Object) 181 10.022 java.lang.invoke.LambdaForm$MH.1318475957.guardWithCatch(Object, Object, Object) 156 8.638 fast run: Stack Trace Sample Count Percentage(%) java.lang.invoke.LambdaForm$DMH.1190900417.invokeStatic_LL_L(Object, Object, Object) 283 13.078 java.lang.invoke.LambdaForm$MH.1318475957.guardWithCatch(Object, Object, Object) 183 8.457 java.lang.invoke.LambdaForm$BMH.1166726978.reinvoke(Object, Object) 40 1.848 If I now look at LambdaForm$MH.514453513.guardWithCatch I see missed inlining due to "already compiled into a big method". So I think it's quite likely a problem with inlining heuristics and I don't see anything pointing in another direction.
27-03-2014
Marcus, I linked 3 issues that I identified when investigating performance of octane.richards and that I intend to get reviewed and pushed for 9.
15-01-2014

Relates :	JDK-8031754 - Type speculation should favor profile data from outermost inlined method
Relates :	JDK-8031752 - Failed speculative optimizations should be reattempted when root of compilation is different
Relates :	JDK-8031755 - Type speculation should be used to optimize explicit null checks