ADDITIONAL SYSTEM INFORMATION :
Mac mini (2018, Inter Core i5, macOS Big Sur 11.1).
Problem reproduce with Java versions 8, 11, 15 and 16 (Oracle JDK, AdoptOpenJDK, Amazon Corretto).

A DESCRIPTION OF THE PROBLEM :
Modal dialog gets hidden behind it's parent frame.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run test code on macOS Big Sur.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Modal dialog remains in from of it's parent frame.
ACTUAL -
Modal dialog gets hidden behind it's parent frame.

---------- BEGIN SOURCE ----------
import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JFrame;

public class Test {
  public static void main(String[] args) {
    JFrame frame = new JFrame();
    frame.getContentPane().setLayout(new FlowLayout());
    JButton button = new JButton("Open modal dialog");
    button.addActionListener(e -> actionOpenDialog(frame));
    frame.getContentPane().add(button);
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
    frame.setSize(600, 450);
    frame.setLocationRelativeTo(null);
    frame.setVisible(true);
  }

  private static void actionOpenDialog(JFrame frame) {
    JDialog dialog = new JDialog(/* parent */ frame, /* modal */ true);
    dialog.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
    dialog.setSize(300, 200);
    dialog.setLocationRelativeTo(dialog.getParent());
    dialog.setVisible(true);
  }
}
---------- END SOURCE ----------

FREQUENCY : always