This bug originated from the following report: http://mail.openjdk.java.net/pipermail/serviceability-dev/2013-August/011318.html
That report has a reproducer that causes attach to stop working on OS X. The reason is that the process ends up with a signal mask that blocks SIGQUIT:
- The VM sets the signal mask for all threads (except the internal VMThread) to mask out SIGQUIT using pthread_sigmask(). So the process will still respond to SIGQUIT, but only one thread.
- The verifier code calls setjmp() to save the calling context. On OS X this also saves the signal mask.
- The example code causes a verification error somewhere which causes the verifier to call longjmp().
- longjmp will restore the signal mask using sigprocmask() which sets the signal mask for the _process_.
- Now the process has a signal mask that masks out SIGQUIT and attach stops working.
This only happens on OS X because setjmp/longjmp does not save and restore the signal mask on other platforms. There are functions _setjmp/_longjmp on OS X to skip the signal mask save/restore and I think this is what we need to use in the verifier (also need to check other uses of setjmp/longjmp in the JDK).