FULL PRODUCT VERSION :
java version "9"
Java(TM) SE Runtime Environment (build 9+181)
Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 10.0.15063]

A DESCRIPTION OF THE PROBLEM :
Closing a stage does not release it's resources.  If there is a ProgressIndicator in the stage, the animation framework will stay connected to this and the hierarchy will not be eligible for garbage collection.

When a Dialog closes, its close() call stack clears the scene, which allows components to clean up their resources automatically.

Java 8U144 behaves the same.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
I used the NB profiler to test the memory.

Run the program, opening up a dialog and then closing it does not cause the ProgressIndicator objects to stay in memory.  Opening and then closing the stage does.  Looking in the profiler, you will see that the animator is the reference chain.

I left a workaround in the code (that is commented out) which clears the scene on close.  This eliminates the leak.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
No leaks
ACTUAL -
Objects are not eligible for GC.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
/*
 * To change this license header, choose License Headers in Project Properties.
 * To change this template file, choose Tools | Templates
 * and open the template in the editor.
 */
package javafxmemoryleak;

import javafx.application.Application;
import javafx.event.ActionEvent;
import javafx.event.EventHandler;
import javafx.scene.Scene;
import javafx.scene.control.Alert;
import javafx.scene.control.Button;
import javafx.scene.control.ProgressIndicator;
import javafx.scene.layout.BorderPane;
import javafx.scene.layout.HBox;
import javafx.scene.layout.StackPane;
import javafx.stage.Stage;

/**
 *
 * @author ocMAClaassen
 */
public class JavaFXMemoryLeak extends Application {

	@Override
	public void start(final Stage primaryStage) {
		Button btn1 = new Button();
		btn1.setText("Open Dialog");
		btn1.setOnAction(new EventHandler<ActionEvent>() {

			@Override
			public void handle(ActionEvent event) {
				openDialog(primaryStage);
			}
		});
		Button btn2 = new Button();
		btn2.setText("Open Stage");
		btn2.setOnAction(new EventHandler<ActionEvent>() {

			@Override
			public void handle(ActionEvent event) {
				openStage(primaryStage);
			}
		});

		HBox root = new HBox();
		root.getChildren().add(btn1);
		root.getChildren().add(btn2);

		Scene scene = new Scene(root, 300, 250);

		primaryStage.setTitle("Memory Test");
		primaryStage.setScene(scene);
		primaryStage.show();
	}
	private void openDialog(Stage primaryStage) {
		StackPane root = new StackPane();
		ProgressIndicator pi = new ProgressIndicator();
		pi.parentProperty().addListener((obs, oldVal, newVal) -> System.out.println("Parent: " + newVal));
		pi.sceneProperty().addListener((obs, oldVal, newVal) -> {
			System.out.println("Dialog Scene: " + newVal);
		});
		root.getChildren().add(pi);

		Alert dialog = new Alert(Alert.AlertType.INFORMATION);
		dialog.getDialogPane().setContent(root);

		dialog.initOwner(primaryStage);
		dialog.showAndWait();

	}
	private void openStage(Stage primaryStage) {
		BorderPane root = new BorderPane();
		ProgressIndicator pi = new ProgressIndicator();
		pi.parentProperty().addListener((obs, oldVal, newVal) -> System.out.println("Parent: " + newVal));
		pi.sceneProperty().addListener((obs, oldVal, newVal) -> {
			System.out.println("Scene: " + newVal);
		});
		root.centerProperty().set(pi);

		final Stage stage = new Stage();
		Button btn = new Button();
		btn.setText("Stage Close");
		btn.setOnAction(new EventHandler<ActionEvent>() {

			@Override
			public void handle(ActionEvent event) {
				stage.close();
				//Workaround
//				stage.getScene().setRoot(new StackPane());
			}
		});

		root.bottomProperty().set(btn);

		Scene scene = new Scene(root);
		stage.setScene(scene);

		stage.showAndWait();

	}

	/**
	 * @param args the command line arguments
	 */
	public static void main(String[] args) {
		System.out.println("java.version = " + System.getProperty("java.version"));
		System.out.println("java.runtime.version = " + System.getProperty("java.runtime.version"));
		launch(args);
	}

}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
When dialogs close, a bogus is set on the scene.   So, imitate this.
stage.getScene().setRoot(new StackPane());
(Heavyweight Dialog has: scene.setRoot(DUMMY_ROOT);)