GTestWrapper.java is a simple wrapper for calling the Hotspot GTests. Notice the calling the HotSpot GTests manually from the command line works perfectly on Alpine/Musl. The problem with the Java wrapper in GTestWrapper.java is as follows:

1. GTests are executed by calling images/test/hotspot/gtest/server/gtestLauncher. "gtestLauncher" is a slim launcher which is dynamically linked against a special version of libjvm.so which contains the GTests. This special libjvm.so is located in the same directory like "gtestLauncher" and "gtestLauncher" is linked with "-rpath" against this special version:

# ldd /priv/output/images/test/hotspot/gtest/server/gtestLauncher
	/lib/ld-musl-x86_64.so.1 (0x7f01e39ac000)
	libjvm.so => /priv/output/images/test/hotspot/gtest/server/libjvm.so (0x7f01e228e000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f01e39ac000)

2. If running with Musl, RequiresSetenv() in java.base/unix/native/libjli/java_md_solinux.c returns "true". This means that the launcher will inject "LD_LIBRARY_PATH" which points to the "well-known" JDK paths into the environment before starting the VM. See the comments in "java_md_solinux.c" for why this happens.

3. In Musl, the search order for resolving dependent libraries is different from Glibc. Glibc uses  DT_RPATH (set by '-rpath') -> LD_LIBRARY_PATH -> system locations. Musl, on the other hand uses LD_LIBRARY_PATH -> DT_RPATH (set by '-rpath') -> system locations (see http://www.openwall.com/lists/musl/2014/03/28/6).

4. GTestWrapper.java executes "gtestLauncher" with the help of "jdk.test.lib.process.ProcessTools.executeCommand()" which in turn uses ProcessBuilder. ProcessBuilder by default uses the default environment for executing "gtestLauncher" but in the case of Musl, this environment was augmented with LD_LIBRARY_PATH pointing to the current JDK.

5. Because of Musl's different resolving order, LD_LIBRARY_PATH takes precedence over the DT_RPATH setting in "gtestLauncher" and libjvm.so will be loaded from the default JDK location which doesn't contain the required GTests.

There are two ways to fix this:

1. (Simple) Fix the test to not use "ProcessTools" but instead use ProcessBuilder directly and remove LD_LIBRARY_PATH from the environment before executing "gtestLauncher".

2. (Advanced) Avoid setting LD_LIBRARY_PATH in launcher. As a consequence we would have to link all shared libraries which are linked against libjvm.so with '-rpath' pointing relatively to the libjvm (e.g. -rpath $ORIGIN/server). As you can see, this hardcodes "server" into DT_RPATH. I don't know if it is still possible to have two configurations (e.g. 'server' and 'client' in one JDK, but even then we could encode both paths into the DT_RPATH).

The problem with the current setup is that it is not easily possible to launch a java process from within a different version of java without changing the environment because of the environment pollution with LD_LIBRARY_PATH.