JDK-8207812 : Implement Dynamic CDS Archive
  • Type: Enhancement
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 11
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2018-07-18
  • Updated: 2019-10-21
  • Resolved: 2019-05-17
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 13
13 b22Fixed
Related Reports
Blocks :  
Blocks :  
CSR :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Sub Tasks
JDK-8226529 :  
Description
Support dynamic archiving classes (including all loaded application classes and library classes that are not present in the base layer archive) as part of a Java application execution:

 - Improve the usability of application class data sharing (AppCDS). Eliminate the need for users to do trial runs to create a classlist for each application.
 - Support both builtin class loaders and customized class loaders effectively and uniformly.

Dynamic archiving for a given application can done at the first execution of the application or when requested by users via command-line option, -XX:ArchiveClassesAtExit=<dynamic archive>. For example:

  % bin/java -XX:ArchiveClassesAtExit=<dynamic archive> -cp hello.jar HelloWorld

Currently, the command line option approach is being implemented.

* The dynamically genarated archive is created on top of the default system archive packaged with the running JDK binary. A separate top-layer archive file is generated for each different Java application. User can specify the application's archive name/path using the -XX:ArchiveClassesAtExit=<dynamic archive> option.

* If desired by user, the system archive can be ignored and all system classes (including the ones in the system archive) loaded during the application execution are archived into the dynamically created archive file. In that case, only one archive (the dynamic archive) is in effect. This behavior can be achieved via a new command-line option, and can be implemented after the initial integration. 

* As the dynamically created top layer archive has dependency to the base archive (e.g. pointers to base layer data), the CRC values of the base archive header and all shared spaces are recorded in the top layer. At runtime, when the dynamic archive is mapped in, all recorded CRC values are compared with the currently mapped base archive CRC values. If any of the CRC value mismatches, the dynamic archive is disabled without affecting the use of the currently used base archive. Using CRC values for base archive dependency checking is more robust than the file name, size and timestamp checks.

* The memory lay-out in the top-layer dynamic archive is similar to the existing static archive that's created using a classlist. Metadata is separated into following three spaces. No 'md' or 'od' space needed.

 - rw space: Read-writable data
 - ro: Read-only data
 - mc: Trampoline

The shared spaced are mapped individually at runtime. The 'ro' space is mapped as read-only for sharing.

* Currently archiving (copying and relocating metadata) is done at the end of the application execution and before VM exiting. If a user defined class loader and associated classes are unloaded, those classes are not archived. Following is a solution for user defined class loaders that is planned after the initial integration:

Copying/relocating of the class metadata can be done immediately after loading a requested class (or before unloading). 

* All loaded application classes and library classes (excluding the ones already included in the base layer and the ones that cannot be archived) are archived in the top-layer. 

* Only class metadata is archived in the top layer currently. Archiving java heap objects in the top-layer dynamically requires more investigations and is not supported in initial implementation. 

* Relocation of the archived java heap objects from the base layer (the default CDS archive) is done at dynamic archiving time, if the narrow oop encoding mode is different from the recoded mode in the base archive or if the archived heap regions are out of the current java heap range. As an optimization, the relocated java objects can be saved in the dynamic archive for subsequent use. 



Comments
[~minqi] AppCDS archive contains system classes which are in the base (default) archive as well as the app class (in the HelloWorld test, only the Hello class). The dynamic CDS archive contains only the app class since other classes already exist in the base (default) archive. Therefore, during AppCDS archive dumping, it needs to process much more classes than dumping a dynamic CDS archive.
08-05-2019

[~jiangli] Regarding the slower startup time with AppCDS archive comparing with base (default) archive, it is due to the VerifySharedSpaces flag will be enabled (set to true) if the SharedArchiveFile is non-null. With VerifySharedSpaces flag enabled, FileMapHeader::validate() will computer the CRC. I've run the tests with a recent JDK 13 nightly build without the dynamic archive changes and observed the same trend. After disabling VerifySharedSpaces (-XX:-VerifySharedSpaces), I got startup time with AppCDS archive as follows: 0.043052859, which is slightly better than with base (default) archive. Claes already has a review out to address the VerifySharedSpaces issue (8221478: Disable VerifySharedSpaces by default).
08-05-2019

@Calvin: Archive creation (sec.) - average of 50 iterations Dynamic CDS dumping 0.051834211 AppCDS dumping 0.655710242 Why is the above data so different?
08-05-2019

Some investigations may be useful to find out why using only the base (default) archive outperforms the single static AppCDS archive and the dynamic archive (default+top) (data copied from the above comment): base archive 0.046085124 Dynamic CDS 0.057627957 AppCDS 0.056631176 The 'AppCDS' archive contains all archived data + HelloWorld class related data. It should perform better than using the default CDS archive. What was the java heap size used during the dumping and runtime for all above cases (except the the default CDS archive dumping)? It's also interesting to see that dynamic dumping takes less time than both 'AppCDS' runtime and 'Dynamic CDS' runtime: Dynamic CDS 0.057627957 AppCDS 0.056631176 Dynamic CDS dumping 0.051834211 <<< Comparing 'AppCDS' runtime with 'Dynamic dumping', the differences are: - HelloWorld is loaded from JAR during 'Dynamic dumping' - Data are copied around during 'Dynamic dumping' I would expect to see a better 'AppCDS' result in the above comparison. Could there be a bug somewhere?
08-05-2019

HelloWorld startup time (sec.) - average of 50 iterations base archive 0.046085124 Dynamic CDS 0.057627957 AppCDS 0.056631176 Archive creation (sec.) - average of 50 iterations Dynamic CDS dumping 0.051834211 AppCDS dumping 0.655710242 sizes of archive (bytes): 12288000 base-hello.jsa (AppCDS archive) 20480 top-hello.jsa (dynamic archive) 12275712 classes.jsa (default CDS archive) Processor: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz O/S: Oracle Linux 7.1 64-bit
07-05-2019

base archive - run the benchmark using only the default CDS archive (classes.jsa) that is part of the JDK. AppCDS - run the benchmark once to create a classlist, create a CDS archive using this classlist with -Xshare:dump, and then run the benchmark using this CDS archive. Dynamic CDS - run the benchmark once with the -XX:ArchiveClassesAtExit=<archive> option to generate a dynamic CDS archive, and then run the benchmark using the dynamic CDS archive. Startup time (sec.) ��� average of 50 iterations application lambda-noop netty noop spring AppCDS 0.067363447 0.525411107 0.05759694 1.947803709 base archive 0.066861514 0.828331755 0.057056222 2.405158521 Dynamic CDS 0.077101064 0.609774081 0.066887995 1.998338412 Footprint (KB) ��� 1 iteration application lambda-noop netty noop spring AppCDS 32052 66908 30596 173808 base archive 35832 83256 35128 143784 Dynamic CDS 37240 71568 37048 187944 Processor: T7610 2 x E5-2630 v2 @ 2.60GHz O/S: Linux 64-bit
07-05-2019

Comments from Ioi: I am not sure if -XX:SharedDynamicArchiveFile is what we want for the long term. I think we would support 1 or 2 levels of archives. Each archive might be dump "statically" using a class list, or "dynamically" as the side effect of running an app (or via jcmd). So calling the top archive "SharedDynamicArchiveFile" seems restrictive. If we already have 2 archives, base.jsa and top.jsa, it really doesn't matter how they are created. Also, the top.jsa should have recorded its dependency (i.e., it depends on /full/path/to/base.jsa). So in fact we can reuse the existing option -XX:SharedArchiveFile without introducing any new options. # implicitly uses base.jsa as recorded in top.jsa java -XX:SharedArchiveFile=top.jsa -cp hello.jar HelloWorld
21-01-2019

Copying to here for broader discussion before JEP is submitted to the open: Use the existing -Xshare:dump/on/auto to control both static (existing dumping method using a static class list) and dynamic dumping. Following are the details. -Xshare:dump =========== 1) If a classlist is explicitly specified in the command line Create an archive using the classlist statically. This is the existing static dumping method. Application specified (if it is specified) on the command line is ignored and not run. Example: java -Xshare:dump -XX:SharedClassListFile=clist -cp hello.jar HelloWorld 2) If no classlist is explicitly specified in the command line 2.1) If no application is specified Create an archive using the default classlist statically. This is the existing static dumping method. Example: java -Xshare:dump 2.2) If an application is specified Create an archive dynamically based on the context of the application execution. This is the new dumping method (dynamic archiving). Example: java -Xshare:dump -cp hello.jar HelloWorld -Xshare:auto =========== 1) If -XX:SharedDynamicArchiveFile=<d_archive> is not explicitly specified in the command line The runtime behavior is the same as today's behavior. 1.1) If -XX:SharedArchiveFile=<s_archive> is specified The specified archive is used if it can be mapped and validated successfully 1.2) If no -XX:SharedArchiveFile=<s_archive> is specified The default CDS archive packaged with the JDK binary is used 2) If -XX:SharedDynamicArchiveFile=<d_archive> is specified in the command line The runtime uses two archives. 2.1) If -XX:SharedArchiveFile=<s_archive> is specified Both 's_archive' and 'd_archive' are used if they can be mapped and validated successfully. Example: java -XX:SharedArchiveFile=base_archive.jsa -XX:SharedDynamicArchiveFile=app_archive.jsa -cp hello.jar HelloWorld 2.2) If -XX:SharedArchiveFile=<s_archive> is not specified The default CDS archive and the 'd_archive' are used if they can be mapped and validated successfully. Example: java -XX:SharedDynamicArchiveFile=app_archive.jsa -cp hello.jar HelloWorld -Xshare:on ======== The -Xshare:on should follow the same rules that are described for -Xshare:auto in the above to use the archives. 1) If -XX:SharedDynamicArchiveFile=<d_archive> is not explicitly specified in the command line The runtime behavior is the same as today's behavior. 1.1) If -XX:SharedArchiveFile=<s_archive> is specified Error is reported if the specified archive cannot be used. 1.2) If no -XX:SharedArchiveFile=<s_archive> is specified Error is reported if the default CDS archive packaged with the JDK binary cannot be used. 2) If -XX:SharedDynamicArchiveFile=<d_archive> is specified in the command line The runtime uses two archives. 2.1) If -XX:SharedArchiveFile=<s_archive> is specified Error is reported for all following cases: - s_archive cannot be used (d_archive is disabled automatically in this case) - s_archive can be used, but d_archive cannot be used Example: java -XX:SharedArchiveFile=base_archive.jsa -XX:SharedDynamicArchiveFile=app_archive.jsa -cp hello.jar HelloWorld 2.2) If -XX:SharedArchiveFile=<s_archive> is not specified Error is reported if for all following cases: - The default CDS archive cannot be used (d_archive is disabled automatically in this case) - The default CDS archive can be used, but d_archive cannot be used Example: java -XX:SharedDynamicArchiveFile=app_archive.jsa -cp hello.jar HelloWorld The design described above provides a flexible solution that adds the dynamic archiving using the existing command-line options (users are already familiar with). It is backward compatible and simple to use.
21-01-2019

Updated the RFE description section to remove 'keeping alive' for custom class loader. Based on the feedback from design review, the initial integration of dynamic archiving will not go with 'keeping alive'. Instead, 'extra copying' will be used for classes loaded by custom class loader. The 'extra copying' approach will be implemented as a follow up step.
20-01-2019

The following was removed from the RFE description section and moved here for reference purpose. In the first release, default location for dynamic archive is not provided. =================================================== Following are the supported application to archive mappings: Main_class ��� main_class_name.jsa -jar JAR_file ��� jar_name.jsa -m Main_module ��� main_module_name.jsa By default, a JSA file using the above name mapping scheme is created in the JDK lib/server directory (or bin/server directory for Windows). For example: % bin/java -Xshare:dynamicDump -cp hw.jar HelloWorld Dumps: ./lib/server/HelloWorld.jsa.
20-01-2019

The suggested fix in JDK-8211723 should be integrated before the integration of dynamic archiving implementation. Testing can be done to make sure incompatible class path change invalidates the top layer dynamic archive at runtime (the dynamic archive should not be mapped and used).
18-01-2019

The tasks will be tracked by following RFEs: This RFE: Support builtin class loaders with dynamic dumping JDK-8192921: Support custom class loader with dynamic dumping
20-07-2018