The problem seems to be in the grab + modality. When a modal dialog is shown, we expect no blocked windows to receive focus and key events. However, F10 is handles separately, so it is retargeted from the dialog to its parent frame and a menu is shown. After this menu can't hidden as no ungrab event is generated (they all are blocked by the modal dialog), and at the same time all the events are grabbed by menu and the X server looks locked.
To fix the problem we need to track how F10 is handled. After we find a frame with menubar that should receive this key event, we should first check if it is blocked and show menu only if it isn't.