JDK-8023720 : (hotspot) setjmp/longjmp changes the process signal mask on OS X
  • Type: Bug
  • Component: hotspot
  • Sub-Component: svc
  • Affected Version: hs24,hs25
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: os_x
  • Submitted: 2013-08-26
  • Updated: 2014-02-12
  • Resolved: 2013-08-29
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7 JDK 8 Other
7u65Fixed 8Fixed hs25Fixed
Related Reports
Cloners :  
JDK-8023786 - (jdk) setjmp/longjmp changes the process signal mask on OS X
Relates :  
JDK-8026385 - [macosx] (awt) setjmp/longjmp changes the process signal mask on OS X
Relates :  
JDK-8028571 - Need test to cover JDK-8023720
Description
This bug originated from the following report: http://mail.openjdk.java.net/pipermail/serviceability-dev/2013-August/011318.html

That report has a reproducer that causes attach to stop working on OS X. The reason is that the process ends up with a signal mask that blocks SIGQUIT:

- The VM sets the signal mask for all threads (except the internal VMThread) to mask out SIGQUIT using pthread_sigmask(). So the process will still respond to SIGQUIT, but only one thread.
- The verifier code calls setjmp() to save the calling context. On OS X this also saves the signal mask. 
- The example code causes a verification error somewhere which causes the verifier to call longjmp().
- longjmp will restore the signal mask using sigprocmask() which sets the signal mask for the _process_.
- Now the process has a signal mask that masks out SIGQUIT and attach stops working.

This only happens on OS X because setjmp/longjmp does not save and restore the signal mask on other platforms. There are functions _setjmp/_longjmp on OS X to skip the signal mask save/restore and I think this is what we need to use in the verifier (also need to check other uses of setjmp/longjmp in the JDK).