JDK-8178720 : CRC32C fails on x86 hardware without CLMUL support
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 9
  • Priority: P3
  • Status: Open
  • Resolution: Unresolved
  • CPU: x86
  • Submitted: 2017-04-13
  • Updated: 2020-05-07
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Duplicate :  
Relates :  
Relates :  
Relates :  
Description
This seems to be a day-one bug which was introduced by JDK-8134553 "CRC32C implementations for x86/x64 targets". It wasn't detected until know because there are not many machines which don't support CLMUL and the CRC32 jtreg tests didn't really work.

During our work on JDK-8176580 "[ppc, s390] CRC32C: wrong checksum result in some cases" we fixed the two regression tests  test/compiler/intrinsics/zip/TestCRC32.java and  test/compiler/intrinsics/zip/TestCRC32C.java which finally unveiled the problem on old x86 hardware.

We will now push JDK-8176580 without the fixed regression tests. The new version of the tests are available from the following webrev:

http://cr.openjdk.java.net/~lucy/webrevs/8176580.03/

and they should be pushed together with the fix for this issue.

Running these test on old hardware will produce the following error:

command: main -Xbatch compiler.intrinsics.zip.TestCRC32C -m
reason: User specified action: run main/othervm/timeout=600 -Xbatch compiler.intrinsics.zip.TestCRC32C -m
Mode: othervm [/othervm specified]
elapsed time (seconds): 1.915
configuration:
STDOUT:
testing 1050 cases ...
STDERR:
ERROR: crc = 6f894393, crcReference = 2cdf6e8f
java.lang.Exception: TestCRC32C Error
        at compiler.intrinsics.zip.TestCRC32C.check(TestCRC32C.java:218)
        at compiler.intrinsics.zip.TestCRC32C.test_multi(TestCRC32C.java:280)
        at compiler.intrinsics.zip.TestCRC32C.main(TestCRC32C.java:74)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:563)
        at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:110)
        at java.base/java.lang.Thread.run(Thread.java:844)

JavaTest Message: Test threw exception: java.lang.Exception: TestCRC32C Error
JavaTest Message: shutting down test

If you don't have old hardware available, you can still manually force the error as describe by Lutz in the mail tread at http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2017-April/026029.html and copied here for your convenience:

The CRC32C implementation generates code depending on the CLMUL cpu feature being available. On most machines @Oracle and @SAP, this feature seems to be available ��� except for one of your MacOS test machines.

To gain control over that feature, I have modified just one line in src/cpu/x86/vm/vm_version_x86_64.hpp like this:
     static bool supports_clmul()    { return ((_features & CPU_CLMUL) != 0) && UseNewCode; }

The boolean UseNewCode is a command line parameter we use @SAP to make experimental code switchable during development. With that modification, I can force off the CLMUL feature at will. And voila, here is what I get (linux-x86_64):

   lu0084 PrivSrc/TestCRC> /<some path>/sapjvm_9/bin/java -XX:+UseNewCode TestCRC32C_OpenJDK -m
   updateBytesCRC32C: pclmulqdq = true.
   testing 1050 cases ...
   lu0084 PrivSrc/TestCRC> /<some path>/sapjvm_9/bin/java -XX:-UseNewCode TestCRC32C_OpenJDK -m
   updateBytesCRC32C: pclmulqdq = false.
   testing 1050 cases ...
   ERROR: crc = 6f894393, crcReference = 2cdf6e8f
   iteration 0: offsets[0] = 0  sizes[30]   = 1024
   Exception: java.lang.Exception: TestCRC32C Error
   Exception in thread "main" java.lang.Exception: java.lang.Exception: TestCRC32C Error
           at TestCRC32C_OpenJDK.test_multi(TestCRC32C_OpenJDK.java:306)
           at TestCRC32C_OpenJDK.main(TestCRC32C_OpenJDK.java:74)
   Caused by: java.lang.Exception: TestCRC32C Error
           at TestCRC32C_OpenJDK.test_multi(TestCRC32C_OpenJDK.java:298)
           ... 1 more
   lu0084 PrivSrc/TestCRC>

Output on MacOS is the same. I did not try solaris-x86_64.
Comments
Hi [~srukmannagar], So changed 'Fix Version' to tbd_feature for now. Please change this later if required. Thanks.
27-04-2018

Hi Rahul[~rraghavan], we would add support for CRC32C for the later versions of jdk and it might not be there in JDK 11. Thanks.
27-04-2018

Hi [~srukmannagar], Request your help. Are you planning to get the fix in for this bug task, for JDK 11? If not, please change the 'Fix Version' to tbd_feature / tbd_update. Thank you.
24-04-2018

Hi [~srukmannagar], Request you help here. If you are not planning to complete this 8178720 for jdk10 by 18Jan2018, please defer, change the 'Fix Version' here to maybe '11'. Thanks.
03-01-2018

I reduced the priority to P3 and set the fix version to 10.
19-04-2017

Hi Igor [~iignatyev], I agree with you suggestion. I'll change the priority once the workaround was pushed (if that's fine with you).
18-04-2017

> ILW=performance of CRC32C on x86 machines without CLMUL support, all usages of CRC32C intrinsic, no workaround=MHH=P2 [~zmajo], I'd say new ILW should be ILW=performance of CRC32C; all usages of CRC32C intrinsic on x86 machines without CLMUL support; no workaround=MMH=P3
18-04-2017

Hi Michael, you and your team are I think more familiar with the CRC32C intrinsic than any of us at Oracle. Could you please take a look at this issue? Thank you! Best regards, Zoltan
13-04-2017

I propose to work around this issue by disabling the CRC32C intrinsic on x86 machines without CLMUL support until the intrinsic can be properly fixed. The workaround is JDK-8178723 and includes (1) the code disabling the intrinsic and (2) the tests provided by Lutz Schmidt. After the workaround provided by JDK-8178723 is in place in JDK 9, we can lower the priority of this issue to P2 because: ILW=performance of CRC32C on x86 machines without CLMUL support, all usages of CRC32C intrinsic, no workaround=MHH=P2 This bug can be used to address the "proper" fix for the issue. If finding the proper fix takes more time than expected, we can defer this issue (as the correctness issue at hand has been solved in JDK 9).
13-04-2017

A shorter summary of steps to reproduce the problem: Lutz's webrev: http://cr.openjdk.java.net/~lucy/webrevs/8176580.03/test/compiler/intrinsics/zip/TestCRC32C.java.sdiff.html The failure reproduces on machines without support for the CLMUL instruction. To reproduce the problem, please follow these steps: (1) apply the changes to TestCRC32C.java from Lutz's webrev (2) apply the following change to the hotspot directory: diff --git a/src/cpu/x86/vm/macroAssembler_x86.cpp b/src/cpu/x86/vm/macroAssembler_x86.cpp --- a/src/cpu/x86/vm/macroAssembler_x86.cpp +++ b/src/cpu/x86/vm/macroAssembler_x86.cpp @@ -10630,6 +10630,8 @@ Label L_byteByByte; Label L_exit; + is_pclmulqdq_supported = false; + if (is_pclmulqdq_supported ) { const_or_pre_comp_const_index[1] = *(uint32_t *)StubRoutines::_crc32c_table_addr; const_or_pre_comp_const_index[0] = *((uint32_t *)StubRoutines::_crc32c_table_addr+1);
13-04-2017

ILW=crash, all usages of CRC32C intrinsic, run with -XX:-UseCRC32CIntrinsics=HHM=P1
13-04-2017