JDK-8237467 : jlink plugin to save the argument files as input to jlink in the output image
  • Type: Enhancement
  • Component: tools
  • Sub-Component: jlink
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2020-01-17
  • Updated: 2022-10-10
  • Resolved: 2022-10-03
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 20
20 b18Fixed
Related Reports
CSR :  
JDK-8294485 - jlink plugin to save the argument files as input to jlink in the output image
Relates :  
JDK-8294394 - running jlink in GraalVM must copy libgraal into the output image
Relates :  
JDK-8232080 - jlink plugins for vendor information and command-line options
Description
Using the capabilities added by JDK-8232080, consider my_image produced by jlink as follows:

> export ALL_MODULES=$(java --list-modules | sed 's:@.*::g' | tr '\n' ',')
> jlink --keep-packaged-modules=my_image/jmods --add-modules $ALL_MODULES --output my_image --vendor-version='XyzzyVM 3.14.15' --vendor-bug-url=https://bugs.xyzzy.com/ --add-options='-Dfoo=xyzzy'
> my_image/bin/java -XshowSettings:properties --version 2>&1 | grep -i xyzzy
    foo = xyzzy
    java.vendor.url.bug = https://bugs.xyzzy.com/
    java.vendor.version = XyzzyVM 3.14.15
Java(TM) SE Runtime Environment XyzzyVM 3.14.15 (build 11.0.5.0.50+1-LTS)
Java HotSpot(TM) 64-Bit Server VM XyzzyVM 3.14.15 (build 11.0.5.0.50+1-LTS, mixed mode)

I would expect that when producing my_image2 from my_image, the custom vendor and option overrides would be preserved without having to specify them again to jlink. However, this appears not to be the case:

> my_image/bin/jlink --keep-packaged-modules=my_image2/jmods --add-modules $ALL_MODULES --output my_image2
> my_image2/bin/java -XshowSettings:properties --version 2>&1 | grep -i xyzzy
>

As a concrete example, for GraalVM this means anyone jlink'ing an image from GraalVM that includes jdk.internal.vm.compiler will be surprised when Graal is not used as the default JIT in the created image. Until this issue is resolved, the following flags need to be explicitly added to the jlink command line:

--add-options='-XX:ThreadPriorityPolicy=1 -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCIProduct -XX:-UnlockExperimentalVMOptions' \
    --add-modules 'java.base,jdk.internal.vm.ci,jdk.internal.vm.compiler'

One complication here is the relationship between command line options (e.g. EnableJVMCI and UseJVMCICompiler) and internal modules (jdk.internal.vm.ci and jdk.internal.vm.compiiler). That is, making changed values for EnableJVMCI and UseJVMCICompiler sticky requires making inclusion of jdk.internal.vm.ci and jdk.internal.vm.compiiler sticky.
Comments
Changeset: 4f44fd63 Author: Doug Simon <dnsimon@openjdk.org> Date: 2022-10-03 16:24:43 +0000 URL: https://git.openjdk.org/jdk/commit/4f44fd63080d40d53a7751ebae93415aeb9b4a47
03-10-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/10445 Date: 2022-09-27 11:12:57 +0000
27-09-2022
The CopyFilesPlugin issue has been split out to JDK-8294394.
26-09-2022
The --save-jlink-argfiles proposal looks good btw.
23-09-2022
libgraal.so belongs to jdk.internal.vm.compiler in the sense that libjvm.so belongs to all the C++ sources in src/hotspot. That is, jdk.internal.vm.compiler is compiled to produce libgraal.so but after that, there's no need to keep jdk.internal.vm.compiler around. While improved jlink execution would be great in general, packing libgraal.so as a module still adds extra complexity to the GraalVM build process. That may be unavoidable. What's more, it also means the footprint of libgraal.so is paid for twice in a GraalVM JDK; once in lib/libjvmcicompiler.so and again packaged in a jmod file. While we're working on reducing the size of libgraal.so, it will take a while yet and it would be unfortunate to penalize the GraalVM JDK download size in the meantime.
23-09-2022
The build time is a separate issue and there could be ways to improve the tooling. If libgraal.so belongs to jdk.internal.vm.compiler, I would think the build can create two packaged modules (1) jdk.internal.vm.compiler.jmod for GVM1 and (2) jdk.internal.vm.compiler.jmod for GVM2 with libgraal.so. Or create a new named module would also work (of course need to create module-info.java) Then specify a different module path to create GVM1 and GVM2 images. This approach seems to work cleanly with the current jlink. Then we can work on improving the link time.
23-09-2022
The problem with that approach is that it complicates and slows down the GVM build process. Currently, there is a single jlink step (to build GVM1). What you are proposing means not only the need for a second jlink step but also packaging a single native lib into a module. I don't think the latter is even possible: > find libgraal/ libgraal/ libgraal/lib libgraal/lib/libjvmcicompiler.so > jmod create --class-path=libgraal --target-platform=linux-amd64 libgraal.jmod Error: module-info.class not found java.io.IOException: module-info.class not found Ideally, we would post-process GVM1/jmods/java.base.jmod to add libjvmcicompiler.so to it (as a sibling of lib/libjava.so). But the jmod command does not support updating .jmod files as far as I know.
23-09-2022
I think --save-argfiles can be implemented as a jlink plugin and independent of jlink's processing of @argfile. The option name may be easily confused that it's saved as the launcher's argfile. `--save-jlink-argfiles` would be clearer and avoid the confusion. The jlink plugin can be in the following form: --save-jlink-argfiles=<filenames> Save the specified argument files that contain the arguments to be prepended to the execution of jlink from the resulting image. <filenames> is a comma-separated path of command-line argument files. --save-jlink-argfiles is a jlink plugin that takes the list of command-line argument files and reads the files and save them in a jdk.jlink resource file. The files listed in the arguments of the --save-argfiles option may or may not be used in that jlink execution. For example, --save-jlink-argfiles=file1,file2 @file1 @wont-save-argfile file1 and file2 are saved but file2 is not used in this jlink execution.
23-09-2022
For the GVM2 files, I would imagine packaging GVM2-specific artifacts in a single module such that GVM2 image can be created using jlink to link all GVM1 modules + GVM2-specific module. Any issues with that approach?
23-09-2022
In addition to persisting certain jlink options, for GraalVM there's also the need to copy artifacts from the runtime image to the jlink output image. The motivating use case is libgraal (deployed as lib/libjvmcicompiler.so on linux). To create libgraal, a base GraalVM image (GVM1) is used to run Native Image. The final libgraal-enabled GraalVM image (GVM2) is produced by copying GVM1 and overlaying it with the libgraal library. That is, in GVM2, there is an artifact (e.g., lib/libjvmcicompiler.so) in GVM2, that was not deployed by jlink. When running jlink in GVM2, this artifact must be copied into the output image so that the output image is a Java runtime that uses libgraal instead of C2 as the top tier compiler. One possible solution is a CopyFilesPlugin which is prototyped and further described at https://github.com/dougxc/jdk/pull/2
22-09-2022
The current proposal from [~mchung] is a --save-argfile option which takes a file an argument. It has the same format as a java Command-Line Argument File[1]. Each argument in the file is applied to the current jlink execution and is also saved in jdk.jlink/jdk/internal/jlink/options in the resulting image. If the resulting image itself contains jlink, then it will read jdk.jlink/jdk/internal/jlink/options and prepend the arguments to the jlink command line args. My design notes: * The value of the --save-argfile option should not have an @ prefix (to avoid clashing with @argfiles currently supported by jlink). * The --save-argfile option must either be a main jlink option as it needs to preprocess the jlinks args (at least, I don't see how this is possible for a jdk.tools.jlink.plugin.Plugin). * There's not reason not to make --save-argfile be a repeating option. [1] https://docs.oracle.com/en/java/javase/17/docs/specs/man/java.html#java-command-line-argument-files
21-09-2022
Yes, I think that will work nicely. I was not aware that $JAVA_HOME/jmods is automatically appended to the --module-path value. As a reference to others (and my future self!), this is explained at https://docs.oracle.com/en/java/javase/13/docs/specs/man/jlink.html: -p or --module-path modulepath Specifies the module path. If this option is not specified, then the default module path is $JAVA_HOME/jmods. This directory contains the java.base module and the other standard and JDK modules. If this option is specified but the java.base module cannot be resolved from it, then the jlink command appends $JAVA_HOME/jmods to the module path.
03-03-2020
If you produce a JDK via jlink that has extra modules, specifying "--keep-packaged-modules $dest/jmods" will copy all modules that you list into the destination jmods directory. If you make sure that the modules end up in the default jmods directory, all dependent modules, including your added ones, will be available without having to specify a --module-path. Does this work for you? % jlink --output test --add-modules java.base,my.module --module-path `pwd` --keep-packaged-modules test/jmods % ls test/jmods java.base.jmod my.module.jmod
03-03-2020
[~bobv] or [~mr] do you have any further thoughts on how to persist --module-path?
03-03-2020
While we could indeed persist --add-modules, we'd also have to persist the --module-path option passed to jlink. While multiple instances of --add-modules compose nicely, I don't think the same applies to --module-path.
05-02-2020
And why can't --add-modules be added to the --persistent-options list.
05-02-2020
I can see how this suggestion solves the problem of options and vendor settings needing to be sticky. However, it doesn't help with the problem of the relationship between certain options and their relationship to specific modules. Once again, consider GraalVM as a concrete example. Anyone jlink'ing an image from GraalVM will get sticky options for EnableJVMCI and UseJVMCICompiler. However, unless they explicitly add the Graal and JVMCI modules to the image, the resulting image will be somewhat broken. Its almost as if we need --persistent-modules in addition to --persistent-options.
05-02-2020
Mark suggested a new option that might solve the reported problem. Do you think this would resolve your issues Doug?? A general solution to this problem would be another option, say `--persistent-options`, whose value would be a list of options to be used in the present invocation of `jlink` and in invocations of the `jlink` tool in the resulting image, and so on, recursively. This could be implemented via a resource file in the `jdk.jlink` module. Then the example in 8232080 would become: $ jlink --add-modules java.base --output /tmp/jre \ --persistent-options="--vendor-version='XyzzyVM 3.14.15' \ --vendor-bug-url=https://bugs.xyzzy.com/ \ --add-options='-Dfoo=xyzzy'" $ /tmp/jre/bin/java -XshowSettings:properties --version 2>&1 | grep -i xyzzy foo = xyzzy java.vendor.url.bug = https://bugs.xyzzy.com/ java.vendor.version = XyzzyVM 3.14.15 OpenJDK Runtime Environment XyzzyVM 3.14.15 (build 14-mr+0-cjdk-81d748451934) OpenJDK 64-Bit Server VM XyzzyVM 3.14.15 (build 14-mr+0-cjdk-81d748451934, mixed mode) $
05-02-2020