The JMX initialization code in com.sun.javafx.jmx.MXExtension in the javafx.graphics module uses the wrong class loader to load the concrete MXExtension subclass. It uses reflection to load this class, but incorrectly assumes that the ClassLoader used to load MXExtension can see the requested class. It should use the CCL rather than the class loader used to load MXExtension itself.
See the attached test programs, which set "javafx.debug=true". The first one uses the default MXExtension (which is shipped with javafx-mx.jar in JDK 8) and the second one uses a custom MXExtension.
In JDK 8, it was possible to have the FX runtime load the MXExtension class by putting javafx-mx.jar (or the path to the custom MXExtension) and jfxrt.jar both on the bootclasspath, but this is no longer possible in 9.
Because JavaFX JMX support is deprecated, and we no longer distribute the javafx-mx.jar file, this is not a serious bug.