FULL PRODUCT VERSION :
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Mac OS Sierra 10.12.6
uname -a output:
Darwin Jessicas-MacBook-Pro.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64

A DESCRIPTION OF THE PROBLEM :
When a modal dialog (A) with parent Frame F creates a new modal dialog (B) with the same parent F, dialog (A) appears on top of dialog (B), even though (B) has focus. Clicking on either dialog corrects focus.

This bug is also present in 8u162:
java version "1.8.0_162-ea"
Java(TM) SE Runtime Environment (build 1.8.0_162-ea-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b01, mixed mode)

Also in 9.0.1:
java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)


REGRESSION.  Last worked in version 8u141

ADDITIONAL REGRESSION INFORMATION: 
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b16, mixed mode)


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Compile the attached source code
2. Launch. Click the "Press the button" button in the application window.
3. A new dialog will open. Click the "Open a new modal dialog" button. A new dialog opens, but it's beneath the previous dialog (dialogs are numbered sequentially so you can easily see which opened first).
4. Click any of the open windows and the last dialog opened will appear on top

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The dialog with focus should appear on top. 
ACTUAL -
The dialog in focus is beneath another dialog.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import javax.swing.*;
import java.awt.*;

public class Main extends JFrame {
    MyPanel panel;

    public Main() {
        setTitle("This is a frame");
        setSize(300, 200);
        panel = new MyPanel(this);
        add(panel);

        setLocationRelativeTo(null);
        setDefaultCloseOperation(EXIT_ON_CLOSE);
    }

    public static void main(String[] args) {
        EventQueue.invokeLater(() -> {
            Main frame = new Main();
            frame.pack();
            frame.setVisible(true);
        });
    }

    private static class MyPanel extends JPanel {

        int dialogCounter = 1;
        final JFrame theParent;

        public MyPanel(JFrame parent) {
            super();
            theParent = parent;
            setPreferredSize(new Dimension(300, 200));
            JButton button = new JButton("Press the button");
            button.addActionListener(e -> showDialog(theParent));

            add(button);
        }

        private void showDialog(Frame parent) {
            JDialog dialog = new JDialog(parent, "This is dialog " + dialogCounter, false);
            setupDialog(dialog);
        }

        private void setupDialog(JDialog dialog) {
            JPanel dialogPanel = new JPanel();
            dialogPanel.setPreferredSize(new Dimension(300, 200));
            dialogPanel.add(new JLabel("Current dialog count: " + dialogCounter++));
            JButton button = new JButton("Open a new modal dialog");
            button.addActionListener(e -> showDialog(theParent));
            dialogPanel.add(button);
            dialog.add(dialogPanel);
            dialog.pack();
            dialog.setModal(true);
            dialog.setVisible(true);
        }
    }
}
---------- END SOURCE ----------