JDK-6704896 : FD_SET usage can cause stack corruption (sol)
  • Type: Bug
  • Component: client-libs
  • Sub-Component: java.awt
  • Affected Version: 7
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: solaris
  • CPU: generic
  • Submitted: 2008-05-20
  • Updated: 2011-01-19
  • Resolved: 2008-09-23
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7
7 b36Fixed
Related Reports
Relates :  
Description
This problem is present in all JDK versions containing the code described below, so backporting is needed.

This applies applies to 32 bit Solaris: both sparc and x86. There is no problem with 64 bit Solaris.

Function SplashEventLoop in ./solaris/native/sun/awt/splashscreen/splashscreen_sys.c uses FD_SET in two places on local array var fds, which only contains 2048 bits. If this code is executed with a descriptor-intensive application on a Solaris system configured to support greater than 2048 descriptors, the FD_SET usage will cause corruption of a caller's stack frame. Either poll must be used instead of select, or else FD_SETSIZE must be made 65536 when this module is compiled.

See the select(3c) Solaris manpage and case PSARC/1997/110 for more details.

The code in question is listed below:

void
SplashEventLoop(Splash * splash) {

    /*      Different from win32 implementation - this loop
       uses select timeouts instead of a timer */
    /* we should have splash _locked_ on entry!!! */

    int xconn = XConnectionNumber(splash->display);

    while (1) {
        int ctl = splash->controlpipe[0];
        fd_set fds[2];
        int n = 0;
        struct timeval tv, *ptv;
        int rc;
        int time;
        int pipes_empty;

        FD_ZERO(fds);
        FD_SET(xconn, fds);
        ^^^^^^^^^^^^^^^^^^^ not sure if xconn
        if (xconn+1 > n)
            n = xconn+1;
        FD_SET(ctl, fds);

Comments
SUGGESTED FIX diff --git a/src/solaris/native/sun/awt/splashscreen/splashscreen_sys.c b/src/solaris/native/sun/awt/splashscreen/splashscreen_sys.c --- a/src/solaris/native/sun/awt/splashscreen/splashscreen_sys.c +++ b/src/solaris/native/sun/awt/splashscreen/splashscreen_sys.c @@ -40,6 +40,7 @@ #include <langinfo.h> #include <locale.h> #include <fcntl.h> +#include <poll.h> static Bool shapeSupported; static int shapeEventBase, shapeErrorBase; @@ -534,40 +535,34 @@ SplashEventLoop(Splash * splash) { /* Different from win32 implementation - this loop - uses select timeouts instead of a timer */ + uses poll timeouts instead of a timer */ /* we should have splash _locked_ on entry!!! */ int xconn = XConnectionNumber(splash->display); while (1) { + struct pollfd pfd[2]; + int timeout = -1; int ctl = splash->controlpipe[0]; - fd_set fds[2]; - int n = 0; - struct timeval tv, *ptv; int rc; - int time; int pipes_empty; - FD_ZERO(fds); - FD_SET(xconn, fds); - if (xconn+1 > n) - n = xconn+1; - FD_SET(ctl, fds); - if (ctl+1 > n) - n = ctl+1; + pfd[0].fd = xconn; + pfd[0].events = POLLIN | POLLPRI; + + pfd[1].fd = ctl; + pfd[1].events = POLLIN | POLLPRI; + errno = 0; if (splash->isVisible>0 && SplashIsStillLooping(splash)) { - time = splash->time + splash->frames[splash->currentFrame].delay + timeout = splash->time + splash->frames[splash->currentFrame].delay - SplashTime(); - if (time < 0) - time = 0; - msec2timeval(time, &tv); - ptv = &tv; - } else { - ptv = NULL; + if (timeout < 0) { + timeout = 0; + } } SplashUnlock(splash); - rc = select(n, fds, NULL, NULL, ptv); + rc = poll(pfd, 2, timeout); SplashLock(splash); if (splash->isVisible>0 && SplashTime() >= splash->time + splash->frames[splash->currentFrame].delay) {
17-06-2008

EVALUATION A testcase to reproduce the problem with splashscreen would be useful.
05-06-2008

SUGGESTED FIX See the suggested fix for java.net.PlainSocketImpl in CR 6670408 for a model.
20-05-2008