JDK-8221706 : Implement Dynamic CDS Archive
  • Type: CSR
  • Component: hotspot
  • Sub-Component: runtime
  • Priority: P3
  • Status: Closed
  • Resolution: Approved
  • Fix Versions: 13
  • Submitted: 2019-03-29
  • Updated: 2019-08-15
  • Resolved: 2019-05-17
Related Reports
CSR :  
JDK-8207812 - Implement Dynamic CDS Archive
Description
Summary
-------

Support dynamic archiving at application runtime when enabled by command line option. The user can then use the dynamic archive in subsequent application run by using an existing command line option.

Problem
-------

With the existing application class-data sharing (AppCDS) feature, the user needs to generate a classlist and then using the classlist to generate a shared archive. Dynamic CDS archive enhances the usability of AppCDS by eliminating the classlist generation step.

Solution
--------

To create a dynamic CDS archive, the user can just run the java application with the following vm option:
 -XX:ArchiveClassesAtExit=<archive name>

To run the same application using the dynamic CDS archive, the user can make use of the existing -XX:SharedArchiveFile=<archive name> vm option.

Specification
-------------

To create a dynamic CDS archive:

    java -XX:ArchiveClassesAtExit=<dynamic archive> -cp <app jar> MyApp

The above command creates a dynamic CDS archive with the default CDS archive as the base archive.
If the default CDS archive does not exist. VM will exit with the following error:

>     ArchiveClassesAtExit not supported when base CDS archive is not loaded

The user could also create a dynamic CDS archive with a specific base archive as follows:

    java -XX:SharedArchiveFile=<base archive> -XX:ArchiveClassesAtExit=<dynamic archive> -cp <app jar> MyApp

Error checks:

 - -Xshare:dump specified with -XX:ArchiveClassesAtExit will result in the following VM error:

>     Error occurred during initialization of VM
>     -XX:ArchiveClassesAtExit cannot be used with -Xshare:dump

 - more than one base archive in the -XX:SharedArchiveFile will result in the following VM error:

>     Error occurred during initialization of VM
>     Cannot have more than 1 archive file specified in -XX:SharedArchiveFile during CDS dumping

 - same archive file specified in -XX:SharedArchiveFile and -XX:ArchiveClassesAtExit will result in the following VM error:

>     Error occurred during initialization of VM
>     Cannot have the same archive file specified for -XX:SharedArchiveFile and -XX:ArchiveClassesAtExit:<archive name>

 - If the base archive does not exist, VM will exit with an error.

If the base archive contains additional path component besides the runtime image and the classpath (-cp), dynamic archive dumping is disabled.
Specifically, if the base archive has appended boot classpath, VM will issue the following warning:

> Dynamic archiving is disabled because base layer archive has appended boot classpath

If the base archive has module path, VM will issue the following warning:

> Dynamic archiving is disabled because base layer archive has module path

To run the application using the dynamic CDS archive and the default CDS (base) archive:

    java -XX:SharedArchiveFile=<dynamic archive> -cp <app jar> MyApp

The above command doesn't specify a base archive. The base archive info, including its name and full path, will be retrieved from the dynamic archive header.

Note that the user could also use the -XX:SharedArchiveFile option for specifying a regular (non-dynamic) CDS archive. Therefore, the specified archive in the -XX:SharedArchiveFile option could be of a regular or a dynamic type. During VM starts up, the specified archive header will be read. If it is of dynamic archive type, the base archive location will be retrieved from the header. It also checks if the default CDS archive flag is set in the header. If the flag is set, it will then obtain the default CDS archive location based on the current run time environment. This is to account for if the default CDS archive file has been moved to a different location after the creation of the dynamic archive. If it is of the regular archive type, then it will work as before as in the existing AppCDS case when the archive is specified at a location.

To run the application using the dynamic CDS archive and a specific base CDS archive:

    java -XX:SharedArchiveFile=<base archive>:<dynamic archive> -cp <app jar> MyApp

The above command is useful if the base archive used for creating the dynamic archive has been moved. Normally, just specifying the dynamic archive should be sufficient since the base archive info can be retrieved from the dynamic archive header.

Error checks:

 - if base archive is specified in the dynamic archive location, VM will issue the following error:

>     Error occurred during initialization of VM       
>     Not a base shared archive:<archive name>

 - if dynamic archive is specified in the dynamic archive location, VM will issue the following error:

>     Error occurred during initialization of VM       
>     Not a top shared archive:<archive name>

 - if base archive not specified, i.e. -XX:SharedArchiveFile=:&lt;dynamic archive&gt;, VM will issue the following error:

>     Error occurred during initialization of VM       
>     Base archive was not specified:<archive name>

 - if dynamic archive not specified, i.e. -XX:SharedArchiveFile=&lt;base archive&gt;:, VM will issue the following error:

>     Error occurred during initialization of VM       
>     Top archive was not specified:<archive name>

 - if more than 2 archive files are specified, VM will issue the following error:

>     Error occurred during initialization of VM       
>     Cannot have more than 2 archive files specified in the -XX:SharedArchiveFile option

If the content of the specified base archive is different from the one used during dynamic archive dump time, VM will report errors such as CRC checksum mismatch.
Comments
Moving to Approved.
17-05-2019
How does this feature support or interact with modules? Module path is supported for the dynamic archive. Same restrictions as in JDK 11 apply. See the following section of JDK 11 release notes for details: https://www.oracle.com/technetwork/java/javase/11-relnote-issues-5012449.html#JDK-8194812 However, we don't support module path yet in the base archive; dynamic archive generation is disabled if the base archive contains module path component. We will investigate supporting it in the future via RFE JDK-8223077. Are the archive format/formats documented somewhere? We never document the format of a CDS archive as the user doesn't need to know the details. The format of a dynamic archive is similar to a base archive with some additional fields in the header portion for storing CRC checksum of the regions of the base archive as well as the base archive name. Where is documentation about using this feature expected? There's a documentation subtask for the corresponding JEP. Contents will be provided in the subtask. Doc team will pick up the content from there and then publish it in the release notes. The following JVM guide will also be updated: https://docs.oracle.com/en/java/javase/12/vm/class-data-sharing.html#GUID-7EAA3411-8CF0-4D19-BD05-DF5E1780AA91
02-05-2019
I'm advancing this request to the intermediate Provisional state as I have some questions I'd like to see answered before the request is finalized. How does this feature support or interact with modules? Are the archive format/formats documented somewhere? Where is documentation about using this feature expected?
01-05-2019
Calvin, ignore the longer lines issue - it is either by design or a tool bug - so no worries. Thank you for clarifying the -XX:ArchiveClassesAtExit behavior and how it matches the existing -XX:SharedArchiveFile. The behavior makes sense to me, I just wanted to make sure it gets documented.
25-04-2019
If the archive file specified in the -XX:ArchiveClassAtExit already exists, it will be overwritten. Same applies to the static archive case with the -XX:SharedArchiveFile option. Regarding the longer command and error messages, they look fine after I've logged in. Maybe that's by design?
25-04-2019
What happens if the dynamic archive already exists and you specified -XX:ArchiveClassesAtExit ?
25-04-2019
Could you please wrap some of the longer command and error messages? They show when editing, but not in the JEP when you are not logged in (sounds like a bug in the tool).
25-04-2019
I mentioned this new feature to my favorite customer, who responded that this sounds very helpful AND - it would be best if the customer could modify their scripts only once, so that they could add a command-line flag that would create the dynamic archive if it does not exist and use it if it does. Otherwise the customer still has to perform multiple steps by changing their scripts. Is it possible to change the the new -XX:ArchiveClassesAtExit to have this meaning?
23-04-2019
This looks great. Thank you for finding this clean path through the command-line arguments with room for future.
19-04-2019
Relaxing it in a follow-up RFE sounds reasonable. Please link the new RFE with this CSR. No other additional comments from me at this point. Thanks.
19-04-2019
Yes, the archive files for the -XX:SharedArchiveFile must be specified in the order of base followed by dynamic archive. Since we are storing the base archive location in the dynamic header, most use cases don't need to specify the base archive together with the dynamic archive in the -XX:SharedArchiveFile option. It is there to accommodate the case when the base archive used for creating the dynamic archive has been moved to a different location. We can definitely relax this restriction as a follow-up RFE if needed.
12-04-2019
Runtime usability improvement suggestions: It is vague if there is an ordering requirement of the base and top dynamic archives in the new -XX:SharedArchiveFile option format. It will be helpful to explicitly specify that. The specification for the new -XX:SharedArchiveFile seems to imply that the archives must be specified in this particular order: <base_archive>:<dynamic_archive>. If that indeed is the case, the requirement can be relaxed with the considerations of the following two aspects: - Strict order requirement in -XX:SharedArchiveFile format is less user friendly - The strict order requirement is not mandated by the design of dynamic archiving, which can already differentiate between base archive and dynamic archive by inspecting header flag Allowing users to specify the base archive and dynamic archive in any order can help improve usability. It would make it easier to use by accepting all following cases as valid usages: 1) -XX:SharedArchiveFile=<base_archive> 2) -XX:SharedArchiveFile=<dynamic_archive> 3) -XX:SharedArchiveFile=<base_archive>:<dynamic_archive> 4) -XX:SharedArchiveFile=<dynamic_archive>:<base_archive> 1) is used when only the base archive is specified 2) can be used with the default CDS archive + dynamic archive. An error can be reported if the default CDS archive does not exist. 3) and 4) are used if the base archive is not the default CDS archive The currently implementation of the dynamic archiving can already support all four cases with small changes to accommodate the different command-line ordering formats in 3) and 4).
12-04-2019
The updated version looks better, thanks. One additional suggestion for the following case: "The user could also create a dynamic CDS archive with a specific base archive as follows: java -XX:SharedArchiveFile=<base archive> -XX:ArchiveClassesAtExit=..." It should also add an error case when a dynamic archive is specified with -XX:SharedArchiveFile. I also have a usability improvement suggestion for the runtime cases. Will provide details later today.
11-04-2019
For the dumping part ( -XX:ArchiveClassesAtExit), it also helps to make things clear by describing the case when the default archive does not exist.
11-04-2019
" - During dumping of a dynamic archive, the CRC values of the base archive header and all shared spaces are recorded in the dynamic archive header. At runtime, when the dynamic archive is mapped in, all recorded CRC values are compared with the currently mapped base archive CRC values. If any of the CRC mismatches, the dynamic archive is not loaded." I'm aware of the above and still freshly remember writing the CRC header checks. So you haven't added any recording of the base archive path location to the dynamic archive since the preliminary open code review, then things are good. To help make it clear, I would suggest adding a description like the following about the base archive specified in -XX:SharedArchiveFile=<base archive>: If the specified base archive content is different from the one used at dynamic dump time, error is reported. It's also helpful to explicitly state that specifying the base archive is optional.
11-04-2019
[~jiangli] Appreciate your quick feedback. Let me try to answer your questions in the same order. - During dumping of a dynamic archive, the CRC values of the base archive header and all shared spaces are recorded in the dynamic archive header. At runtime, when the dynamic archive is mapped in, all recorded CRC values are compared with the currently mapped base archive CRC values. If any of the CRC mismatches, the dynamic archive is not loaded. - Yes, the user does not need to specify the base archive. We already have a test case for it. - 'relocation' means the base archive file has been moved or copied to a different location. I will reword it.
11-04-2019
Here is some early feedback on the CSR, and helpfully it can help with the process. Some clarifications would be needed for the following: "Note that the user could also use the -XX:SharedArchiveFile option for specifying a regular (non-dynamic) CDS archive. Therefore, the specified archive in the -XX:SharedArchiveFile option could be of a regular or a dynamic type. During VM starts up, the specified archive header will be read. If it is of dynamic archive type, the base archive location will be retrieved from the header. It also checks if the default CDS archive flag is set in the header. If the flag is set, it will then obtain the default CDS archive location based on the current run time environment. This is to account for if the default CDS archive has been relocated to a different location after the creation of the dynamic archive. If it is of the regular archive type, then it will work as before as in the existing AppCDS case when the archive is specified at a location." - What happens when the user-specified base archive is different from the one that's recorded in the dynamic archive header? - As the dynamic archive header already records the base archive location, the VM should be able to locate the base archive if one (base archive) is not specified in the command-line. This would be a common use case and simplifying the commend-line option requirement would be useful and improve user experiences. - "This is to account for if the default CDS archive has been relocated to a different location after the creation of the dynamic archive." Clarification is needed for 'relocation' (or probably it should be reworded), as 'relocation' could mean coping the base archive file to a different location (directory, host, etc), or relocating the metadata mapped from the base archive.
10-04-2019