To reproduce this, run the following in non-jigsaw "legacy" mode.
$ unset JIGSAW_HOME
$ gradle --info -PFULL_TEST=true :systemTests:test
The following tests fail to run:
MainLauncherTest
SandboxAppTest
ShutdownHookTest
The error is the same for all of them:
Error: Could not create the Java Virtual Machine.
-Djava.ext.dirs=/localhome/kcr/javafx/9-kcr/jfx/rt/build/sdk/lib is not supported. Use -classpath instead.
This issue was fixed in 9-dev with JDK-8150307, so either the bug was reintroduced or the fix was lost when merging FX 9 into the 9-jake sandbox.