I have a frame with a menu added to the menubar. On clicking a button on the frame, I am opening a file dialog. When the file dialog is visible, I am pressing 'F10'. This has been dispatched to the frame and the menu comes up. But Xserver hangs after that and I have to kill the java process from another machine. It is incorrect that frame's menu responds to F10 when a modal dialog is visible.
This is reproducible on SolSparc10-JDS as well as SuSE10 with JDK6 as well as JDK7-b09. I tried regressing this failure and found that the menu always responds to F10 on JDK5 but the Xserver started hanging only after JDK6-b40. Xserver behaves normally on b30. Lot of fixes have gone into b39 and b40 of JDK6.
I have attached a sample test. Execute it and press the button. A file dialog would come up. Once the file dialog comes, press F10. Xserver will stop responding to user interactions.