JDK-8281075 : Regression ~37% in Renaissance-NaiveBayes on Mac only after JDK-8276112
Type:Bug
Component:hotspot
Sub-Component:compiler
Affected Version:18
Priority:P3
Status:Closed
Resolution:Duplicate
OS:os_x
Submitted:2022-02-01
Updated:2022-05-05
Resolved:2022-02-28
The Version table provides details related to the release that this issue/RFE will be addressed.
Unresolved : Release in which this issue/RFE will be addressed. Resolved: Release in which this issue/RFE has been resolved. Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.
I did a CI build by build tracking of this one where this JDK-8276112 appeared in 18-b24. For some reason it is only observed on Mac.
Comments
OK, yes the variation is generally big with Renaissance, it's unfortunate.
Earlier we were running with:
NaiveBayes -t 1 -i 15 -wi 25 -f 10
You could try that too. Even the variation seems to come and go build to build, some runs are stable, then very bad again.
05-05-2022
hi, @ecaspole,
I ran like you did on a macOS baremetal instance on EC2. I uploaded the log here with jdk17
What I saw is that the stddev is so big. I can't correlate it to JDK-8276998 even though I can see that it is better with that change.
Result "org.renaissance.apache.spark.JmhNaiveBayes.runOperation":
1477.252 ±(99.9%) 61.139 ms/op [Average]
(min, avg, max) = (1190.410, 1477.252, 1799.747), stdev = 180.268
CI (99.9%): [1416.114, 1538.391] (assumes normal distribution)
What is your stdev? do you know why this benchmark has such big variance? I also try to run it directly with 50 iterations. same problem
04-05-2022
Hi [~xliu] in these tests we are running Renaissance 0.13 in the JMH build on x64 Mac Minis.
The JVM flags are:
XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED -XX:-IgnoreUnrecognizedVMOptions
And the JMH flags are:
NaiveBayes -bm avgt -t 1 -w 6 -wi 10 -r 5 -i 10 -f 10
03-05-2022
hello, @ecaspole,
Could you let me know that Renaissance version , MacOS version and hardware platform(Intel x86 or M1)?
02-05-2022
I'm closing this as a dup of JDK-8276998 which should fix this regression again as confirmed by [~ecaspole].
28-02-2022
[~ecaspole] Did you use JDK 18 for last experiment?
Would be interesting to see data for latest JDK (19). I pushed JDK-8276455 Iterative EA in b02. It may offset some regression.
02-02-2022
My fix for JDK-8261137 simply disabled the incorrect box elimination optimization introduced by JDK-8261137 in JDK 17. The regression is therefore only between JDK 17 and 17.0.2 (to which my fix was backported). The plan is to fix and re-enable the optimization with JDK-8276998.
[~whuang], any updates on that?
02-02-2022
ILW = perf regression; seen in one benchmark on Mac only; no workaround = MMH = P3
02-02-2022
If it's a needed fix that is fine, this is just to document it. I will try your suggestion.
01-02-2022
JDK-8276112 fixed incorrect changes - we can't reverse it.
We need to find other solution for this performance "regression".
Also to make sure it is caused by JDK-8276112, please change `false` in next code with `UseNewCode` and run the same build with this flag on and off:
https://github.com/openjdk/jdk18/blob/master/src/hotspot/share/opto/callGenerator.cpp#L686