JDK-8237776 : Shenandoah: Wrong result with Lucene test
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 11-shenandoah,14,15
  • Priority: P1
  • Status: Resolved
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2020-01-23
  • Updated: 2020-11-27
  • Resolved: 2020-02-05
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 14 JDK 15
14 b35Fixed 15Fixed
Related Reports
Relates :  
JDK-8241765 - Shenandoah: AARCH64 need to save/restore call clobbered registers before calling keepalive barrier
Description
One of the tests in the Lucene testsuite fails with a wrong result.

Steps to reproduce:
1. Download Lucene: https://lucene.apache.org/core/downloads.html (tested with 8.4.1)
2. Unpack and setup Ant build according to instructions (https://github.com/apache/lucene-solr/blob/master/lucene/BUILD.txt)
3. Run test:
export JAVA_HOME=/path/to/jdk-under-test
ant beast -Dbeast.iters=2000 -Dtestcase=TestBagOfPostings -Dargs="-XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC"

ant beast runs the test 2000 times. It always failed for me before completing 2000 runs.

The test does not seem to fail with -XX:-TieredCompilation nor does it seem to fail with -Xint, which indicates (but does not prove) that it's related to C1.

It fails in different ways, but always similar in that some result is wrong:

  [beaster] Started J0 PID(23718@localhost).
  [beaster]   2> Jan 23, 2020 6:17:15 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
  [beaster]   2> WARNING: Uncaught exception in thread: Thread[Thread-0,5,TGRP-TestBagOfPostings]
  [beaster]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: maxMergedSegmentMB must be >=0 (got -7.478635795308384E240)
  [beaster]   2> 	at __randomizedtesting.SeedInfo.seed([7CE6ED455350A485]:0)
  [beaster]   2> 	at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:114)
  [beaster]   2> Caused by: java.lang.IllegalArgumentException: maxMergedSegmentMB must be >=0 (got -7.478635795308384E240)
  [beaster]   2> 	at org.apache.lucene.index.TieredMergePolicy.setMaxMergedSegmentMB(TieredMergePolicy.java:161)
  [beaster]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1266)
  [beaster]   2> 	at org.apache.lucene.index.RandomIndexWriter.maybeFlushOrCommit(RandomIndexWriter.java:200)
  [beaster]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:194)
  [beaster]   2> 	at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:111)
  [beaster]   2> 
  [beaster]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBagOfPostings -Dtests.method=test -Dtests.seed=7CE6ED455350A485 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=en-AG -Dtests.timezone=Antarctica/Troll -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  [beaster] [19:17:14.075] ERROR   1.74s | TestBagOfPostings.test <<<
  [beaster]    > Throwable #1: java.lang.AssertionError: expected:<412> but was:<410>
  [beaster]    > 	at org.junit.Assert.fail(Assert.java:88)
  [beaster]    > 	at org.junit.Assert.failNotEquals(Assert.java:834)
  [beaster]    > 	at org.junit.Assert.assertEquals(Assert.java:645)
  [beaster]    > 	at org.junit.Assert.assertEquals(Assert.java:631)
  [beaster]    > 	at org.apache.lucene.index.TestBagOfPostings.test(TestBagOfPostings.java:131)
  [beaster]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  [beaster]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  [beaster]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  [beaster]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  [beaster]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  [beaster]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  [beaster]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  [beaster]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  [beaster]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  [beaster]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  [beaster]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  [beaster]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  [beaster]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  [beaster]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  [beaster]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  [beaster]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  [beaster]    > 	at java.base/java.lang.Thread.run(Thread.java:832)
  [beaster]    > Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=28, name=Thread-0, state=RUNNABLE, group=TGRP-TestBagOfPostings]
  [beaster]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: maxMergedSegmentMB must be >=0 (got -7.478635795308384E240)
  [beaster]    > 	at __randomizedtesting.SeedInfo.seed([7CE6ED455350A485]:0)
  [beaster]    > 	at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:114)
  [beaster]    > Caused by: java.lang.IllegalArgumentException: maxMergedSegmentMB must be >=0 (got -7.478635795308384E240)
  [beaster]    > 	at org.apache.lucene.index.TieredMergePolicy.setMaxMergedSegmentMB(TieredMergePolicy.java:161)
  [beaster]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1266)
  [beaster]    > 	at org.apache.lucene.index.RandomIndexWriter.maybeFlushOrCommit(RandomIndexWriter.java:200)
  [beaster]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:194)
  [beaster]    > 	at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:111)
  [beaster]   2> NOTE: leaving temporary files on disk at: /home/rkennke/src/lucene-8.4.1/build/core/test/J0/temp/lucene.index.TestBagOfPostings_7CE6ED455350A485-001
  [beaster]   2> NOTE: test params are: codec=Asserting(Lucene84): {field=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{}, maxPointsInLeafNode=565, maxMBSortInHeap=7.837812716170918, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@9a5e068), locale=en-AG, timezone=Antarctica/Troll
  [beaster]   2> NOTE: Linux 4.18.0-147.3.1.el8_1.x86_64 amd64/N/A 15-internal (64-bit)/cpus=32,threads=1,free=509967512,total=536870912
  [beaster]   2> NOTE: All tests run in this JVM: [TestBagOfPostings]
  [beaster] 
  [beaster] Tests with failures [seed: 7CE6ED455350A485]:
  [beaster]   - org.apache.lucene.index.TestBagOfPostings.test
Comments
Changeset: ba7d18db Author: Roland Westrelin <roland@openjdk.org> Date: 2020-01-31 14:36:07 +0000 URL: https://git.openjdk.java.net/panama-foreign/commit/ba7d18db
07-02-2020
URL: https://hg.openjdk.java.net/jdk/jdk14/rev/4a87bb7ebfd7 User: roland Date: 2020-02-05 16:33:47 +0000
05-02-2020
Fix request approved
05-02-2020
Fix Request This bug causes incorrect code execution: floating point arguments can be corrupted at a call that goes through the c2i. The bug was introduced with nmethod barriers support for shenandoah in jdk 14 so this is a regression from previous jdk versions. The fix consists in saving extra registers around the runtime call that corrupts fp registers. So it's low risk. Furthermore, the fix is in shenandoah specific code so doesn't affect anything else. We had the fix go through our usual testing. The review thread is: https://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2020-January/028478.html & https://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2020-February/028496.html The webrev is: http://cr.openjdk.java.net/~roland/8237776/webrev.01/
03-02-2020
The problem disappears when passing: -XX:CompileCommand=exclude,org/apache/lucene/util/LuceneTestCase.maybeChangeLiveIndexWriterConfig I suspect the problem might be a miscompilation of that method: org/apache/lucene/util/LuceneTestCase.maybeChangeLiveIndexWriterConfig
28-01-2020
A workaround is -XX:ShenandoahCodeRootsStyle=1, but that is not an option because it disables concurrent class unloading and roots processing wholesale.
27-01-2020