JavaFX uses both X11 and gtk code and gtk3 uses Wayland backend while running Ubuntu 18.04 in Wayland mode. This results in crash in JavaFX. This crash is being done under https://bugs.openjdk.java.net/browse/JDK-8210411 by setting the gtk3 backend to x11 on all systems by setting GDK_BACKEND=x11. This fixes the JavaFX crash.
However, while using the a JFXPanel inside Swing components in Swing-FX interop with GTKLookAndFeel set, the AWT loads the gtk3 first and as the GDK_BACKEND=x11 is not yet set even after the fix for , the gtk3 starts using the Wayland backend. Then while initializing the JFXPanel, JavaFX uses the same gtk library loaded by AWT and this results in crash.
The fix is to set the gtk backend to x11 in AWT as well by setting the GDK_BACKEND=x11. This fixes the Swing-FX interop crash as well.
To overcome this issue while using Swing-FX interop, the GDK_BACKEND=x11 needs to be set in AWT also, so that gtk3 will use x11 backend even when loaded from AWT.