When the StageStyle is EXTENDED JavaFX adds a hidden NSToolbar to the NSWindow (this toolbar positions the stoplight controls). This alters the behavior of the NSWindow minSize and maxSize properties; when either property is set the OS adds a constant to the height. The amount added seems to be the difference between the nominal title bar height (28) and the height of the combined title bar + toolbar area. The exact value isn't important, we can figure it out by setting the minSize or maxSize property and then immediately querying it to see what the OS added.

To reproduce, run the attached test case and press the "Smaller than Minimum" or "Larger than Maximum" buttons and look at the new window height. In the minimum case it will be 316 instead of 300 and in the maximum case it will be 716 rather than 700. Under the hood this is the amount the OS added to the height of the minSize and maxSize properties.

I should add that if the NSWindow's toolbarStyle is updated the NSWindow will be resized and the minSize and maxSize properties will be adjusted by the OS. I'm not sure if the HeaderBar code can change this property on the fly; if so there are more cases to be tested.