JDK-8252730 : jlink does not create reproducible builds on different servers
  • Type: Bug
  • Component: tools
  • Sub-Component: jlink
  • Affected Version: 11.0.8,15,16
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: linux_ubuntu
  • CPU: x86_64
  • Submitted: 2020-09-02
  • Updated: 2024-12-03
  • Resolved: 2020-09-26
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 16
16 b18Fixed
Related Reports
Relates :  
JDK-8241602 - jlink does not produce reproducible jimage files
Relates :  
JDK-8253688 - tools/jlink/JLinkReproducible3Test.java fails with "RuntimeException: jlink producing inconsistent result in modules. Mismatch in modules file"
Description
ADDITIONAL SYSTEM INFORMATION :
Ubuntu 18.04
JDK 15 Build 36 (2020/8/11)
JDK 16 Build 13 (2020/8/26)

A DESCRIPTION OF THE PROBLEM :
Calling (the exact same) jlink with the exact same parameters on different machines and even folder results different output.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
JDK15:

    $ mkdir jdk-15a jdk-15b
    $ tar xzf openjdk-15_linux-x64_bin.tar.gz --directory jdk-15a/
    $ tar xzf openjdk-15_linux-x64_bin.tar.gz --directory jdk-15b/
    $ jdk-15a/jdk-15/bin/jlink --compress=1 --module-path jdk-15a/jdk-15/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output output01
    $ jdk-15b/jdk-15/bin/jlink --compress=1 --module-path jdk-15b/jdk-15/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output output02
    $ md5sum output*/lib/modules
    9ae5ae63df59593a0892c5ed411d7e59  output01/lib/modules
    24b4af31553670799ab286ad1186329e  output02/lib/modules

JDK16:

    $ mkdir jdk-16a jdk-16b
    $ tar xzf openjdk-16-ea+13_linux-x64_bin.tar.gz --directory jdk-16a/
    $ tar xzf openjdk-16-ea+13_linux-x64_bin.tar.gz --directory jdk-16b/
    $ jdk-16a/jdk-16/bin/jlink --compress=1 --module-path jdk-16a/jdk-16/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output output01
    $ jdk-16b/jdk-16/bin/jlink --compress=1 --module-path jdk-16b/jdk-16/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output output02
    $ md5sum output*/lib/modules
    266b05c93ad3858757ec5f194bc7c019  output01/lib/modules
    e27a144d62605719bd8017c75e6aca69  output02/lib/modules


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Same checksum for both lib/modules file.
ACTUAL -
Different cheksums for the lib/modules file.

FREQUENCY : always
Comments
Changeset: 79904c1f Author: Ian Graves <igraves@openjdk.org> Committer: Alan Bateman <alanb@openjdk.org> Date: 2020-09-26 07:01:22 +0000 URL: https://git.openjdk.java.net/jdk/commit/79904c1f
26-09-2020
Ian, one of the first things I would check is if elements in HashSets all have overridden equals/hashCode. The randomness may be due to fallback to identityHashCode.
08-09-2020
Additional Information from Submitter: =========================== It's an Ubuntu 18.04 with an ext4 filesystem. I run every command below on the *same* system but running jlink with JDKs in different folders result different images. Details are below. $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS" $ wget -q https://download.java.net/java/early_access/jdk16/13/GPL/openjdk-16-ea+13_linux-x64_bin.tar.gz $ mkdir jdk-16a jdk-16b $ tar xzf openjdk-16-ea+13_linux-x64_bin.tar.gz --directory jdk-16a/ $ tar xzf openjdk-16-ea+13_linux-x64_bin.tar.gz --directory jdk-16b/ $ jdk-16a/jdk-16/bin/jlink --module-path jdk-16a/jdk-16/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output image-a-default $ jdk-16a/jdk-16/bin/jlink --compress=1 --module-path jdk-16a/jdk-16/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output image-a-compress1 $ jdk-16a/jdk-16/bin/jlink --compress=0 --module-path jdk-16a/jdk-16/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output image-a-compress0 $ jdk-16b/jdk-16/bin/jlink --module-path jdk-16b/jdk-16/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output image-b-default $ jdk-16b/jdk-16/bin/jlink --compress=1 --module-path jdk-16b/jdk-16/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output image-b-compress1 $ jdk-16b/jdk-16/bin/jlink --compress=0 --module-path jdk-16b/jdk-16/jmods --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output image-b-compress0 $ md5sum image-*/lib/modules 7ecb3471c03ba985d7b6ba6e4c087c36 image-a-compress0/lib/modules 93a7a65aada77fa3b439c3de98f5f6bf image-a-compress1/lib/modules 7ecb3471c03ba985d7b6ba6e4c087c36 image-a-default/lib/modules c4999fa6524bffa0578cc9cb861817cb image-b-compress0/lib/modules 574456fa3fbab6d93bf9395c6b15265b image-b-compress1/lib/modules c4999fa6524bffa0578cc9cb861817cb image-b-default/lib/modules $ jdk-16a/jdk-16/bin/java -version openjdk version "16-ea" 2021-03-16 OpenJDK Runtime Environment (build 16-ea+13-521) OpenJDK 64-Bit Server VM (build 16-ea+13-521, mixed mode, sharing) $ jdk-16a/jdk-16/bin/jimage info image-a-compress0/lib/modules Major Version: 1 Minor Version: 0 Flags: 0 Resource Count: 7320 Table Length: 7320 Offsets Size: 29280 Redirects Size: 29280 Locations Size: 140549 Strings Size: 169451 Index Size: 368588 $ jdk-16a/jdk-16/bin/jimage info image-b-compress0/lib/modules Major Version: 1 Minor Version: 0 Flags: 0 Resource Count: 7320 Table Length: 7320 Offsets Size: 29280 Redirects Size: 29280 Locations Size: 141051 Strings Size: 169451 Index Size: 369090 $ jdk-16a/jdk-16/bin/jimage info image-a-compress1/lib/modules Major Version: 1 Minor Version: 0 Flags: 0 Resource Count: 7320 Table Length: 7320 Offsets Size: 29280 Redirects Size: 29280 Locations Size: 167675 Strings Size: 4001049 Index Size: 4227312 $ jdk-16a/jdk-16/bin/jimage info image-b-compress1/lib/modules Major Version: 1 Minor Version: 0 Flags: 0 Resource Count: 7320 Table Length: 7320 Offsets Size: 29280 Redirects Size: 29280 Locations Size: 167673 Strings Size: 4001049 Index Size: 4227310 $ jdk-16a/jdk-16/bin/jimage list --verbose image-a-compress0/lib/modules > image-a-compress0.list $ jdk-16a/jdk-16/bin/jimage list --verbose image-b-compress0/lib/modules > image-b-compress0.list $ head image-a-compress0.list jimage: image-a-compress0/lib/modules Module: java.base Offset Size Compressed Entry 20463334 41 0 META-INF/services/java.nio.file.spi.FileSystemProvider 23945457 574 0 com/sun/crypto/provider/AESCipher$AES128_CBC_NoPadding.class 23948901 574 0 com/sun/crypto/provider/AESCipher$AES128_CFB_NoPadding.class 23943735 574 0 com/sun/crypto/provider/AESCipher$AES128_ECB_NoPadding.class 23950623 574 0 com/sun/crypto/provider/AESCipher$AES128_GCM_NoPadding.class 23947179 574 0 com/sun/crypto/provider/AESCipher$AES128_OFB_NoPadding.class $ head image-b-compress0.list jimage: image-b-compress0/lib/modules Module: java.base Offset Size Compressed Entry 22432183 41 0 META-INF/services/java.nio.file.spi.FileSystemProvider 25914306 574 0 com/sun/crypto/provider/AESCipher$AES128_CBC_NoPadding.class 25917750 574 0 com/sun/crypto/provider/AESCipher$AES128_CFB_NoPadding.class 25912584 574 0 com/sun/crypto/provider/AESCipher$AES128_ECB_NoPadding.class 25919472 574 0 com/sun/crypto/provider/AESCipher$AES128_GCM_NoPadding.class 25916028 574 0 com/sun/crypto/provider/AESCipher$AES128_OFB_NoPadding.class Let me know if you need any further information. I have tested this on an Ubuntu 16.04 box and got very similar results.
07-09-2020
I ran the following on two OEL 7.6 systems (2 jlink command to check the default and compress=1 cases) $ java -version openjdk version "16-ea" 2021-03-16 OpenJDK Runtime Environment (build 16-ea+13-521) OpenJDK 64-Bit Server VM (build 16-ea+13-521, mixed mode, sharing) $ jlink --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output image1 $ jlink --add-modules java.base,jdk.management,jdk.unsupported,jdk.charsets --output image2 --compress 1 $ md5sum image1/lib/modules image2/lib/modules 464b5d31147d95d481fe244a4ca667dd image1/lib/modules 92afde63294be84f68707444d2c736a2 image2/lib/modules The MD 5 hash / output is the same on both system so I'm not seeing the reported issue.
03-09-2020
jlink could not produce reproducible builds before JDK 15 so I don't think there is a regression here. The issues fixed in JDK 15 include JDK-8241602, JDK-8243666, and JDK-8241616 so that builds are at least reproducible on the same system. This bug report is JDKs on different systems. Also it uses --compress=1. It would be useful if the submitter can say if they see the issue when building on the same system, also useful to know if the issue duplicates if they don't use the compression option (as it uses the system zlib so might be different versions on the two systems).It would also be really help to run `jimage extract` on both systems to extract the contents of the generated lib/modules to see if there are any content differences (I'll guess not, and that this is more likely to be something to do with timestamps and/or different versions of zlib).
03-09-2020
After executing jlink on same build extracted in different directories, different MD5 are produced. The issue can be seen on linux versions and not on Windows versions. Following observations can be seen: ========================== JDK 9: Pass JDK 11.0.2: Pass JDK 11.0.8: Fail JDK 14.0.2 : Pass JDK 15 ea 36: Fail JDK 16 ea 11: Fail Both OpenJDK as well as Oracle JDK has this issue.
03-09-2020