Bug ID: JDK-8241071 Generation of classes.jsa with -Xshare:dump is not deterministic

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 15
15 b22Fixed

In the effort of creating a reproducible build, we have so far turned a blind eye towards classes.jsa, since the generated binary is different every time.

The most common causes for this kind of issues are binary formats that contains raw, unsorted dumps of hash maps, and timestamps. Fixing the output so it remains stable upon repeated runs on the same environment makes it possible to verify the consistency of the build. 

In some rare cases, there have been other bugs hiding in non-determinism. It is also good to be sure that the cause for non-determinism in classes.jsa is not such a bug.

I'm currently making a change in the build system, where I'd like to be more sure that I do not affect the content of classes.jsa. So if this is possible to fix relatively soon, I'd be very happy.

----
Review thread: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2020-April/039135.html
----
For debugging determinism issues with classes.jsa, see https://wiki.openjdk.java.net/display/HotSpot/Debugging+CDS+DeterministicDump.java+test+failures

URL: https://hg.openjdk.java.net/jdk/jdk/rev/1cfd2ca0b769 User: iklam Date: 2020-05-05 21:29:41 +0000
05-05-2020
The current fix requires a new native call for the initialization of ImmutableCollections.SALT32L. We can simplify this with a non-writable system property (pending fix of JDK-8243936) http://cr.openjdk.java.net/~iklam/jdk15/8241071-deterministic-cds-archive.v03/
29-04-2020
The following two items need to be fixed: [1] MetaspaceShared::_i2i_entry_code_buffers contain addresses to the code cache, which varies from run to run. It should be zeroed. (The contents will be reinitialized at run-time anyway). [2] java/util/ImmutableCollections.SALT32L should be changed to a deterministic value when running -Xshare:dump. This can be done by modifying ImmutableCollections::SALT32L and ImmutableCollections::REVERSE after this class has been initialized. Otherwise it affects the order of the objects in the shared heap; also the list of packages passed to JVM_DefneModule would have a random order, causing the C++ Symbols to be allocated in random order. The callstack is: C++ JVM_DefineModule(..., const char* const* packages, jsize num_packages) Java java.lang.Module.defineModule0() Java java.lang.Module.<init>(Module.java:136) Java java.lang.System$2.defineModule(System.java:2215) Java jdk.internal.module.Modules.defineModule(Modules.java:82) Java jdk.internal.module.ModuleBootstrap.boot(ModuleBootstrap.java:231) Java java.lang.System.initPhase2(System.java:2070)
17-03-2020
We noticed the non-deterministic issue with .jsa as well. It probably should be handled as a higher priority item. I looked a bit into it a while back, but haven't fully investigated the issue. One of the easier item is the timestamps recorded for the JAR files from the class paths. The timestamp check for runtime and dump time path compatibility is unreliable, particularly with cloud environment. The path validation can fail even when there is no update to the JAR file(s), as the timestamps from the copied JAR file(s) are different. The .jsa dump time and runtime compatibility can be achieved via proper build/packaging/release process. We implemented a new packaging scheme to help ensure the compatibility between the JAR file and CDS archive. I will create a RFE and contribute it to OpenJDK later. The other items involved would be more difficult. Randomness can be intentional in some cases.
16-03-2020

Blocks :	JDK-8243936 - NonWriteable system properties are actually writeable
Relates :	JDK-8245335 - [TESTBUG] DeterministicDump.java fails with release JVM
Relates :	JDK-8335449 - runtime/cds/DeterministicDump.java fails with File content different at byte ...
Relates :	JDK-8275867 - Make ImmutableCollections deterministic if running with TraceBytecodes/StopInterpreterAt
Relates :	JDK-8244536 - cds/DeterministicDump.java failed: File content different
Relates :	JDK-8244489 - Zero and minimal VM build failure after JDK-8241071 (MetaspaceShared::symbol_space_alloc is undefined)