FULL PRODUCT VERSION :
java version  " 1.7.0_21 " 
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)


FULL OS VERSION :
Linux xxx.xxx.xxx 2.6.18-128.1.6.el5 #1 SMP Tue Mar 24 12:05:57 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
We have native code in our JVM. The native code uses signals. It attempts to call sigaction() where the signal >= 0x20.

We LD_PRELOAD the libjsig.so library as documented.

When the native code attempts to sigaction(0x2B) i.e. signal 0x2B, the JVM crashes.

The problem is with the macro

#define MAXSIGNUM 32
#define MASK(sig) ((unsigned int)1 << sig)

In OpenJDK source file jsig.c.

MASK() appears to assume that if sig>=0x20 then the result will be zero. But this is not correct. When shifting a 32-bit integer by a variable amount, the shift amount is masked by 0x1F before the shift happens.

So ((unsigned int)1 << 0x2B) is the same as ((unsigned int)1 << 0x0B) and this results in an attempt to access beyond the end of the array sact[].


THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Write a JNI function that calls sigaction(0x2B) and run it while libjsig.so is LD_PRELOADed.

EXPECTED VERSUS ACTUAL BEHAVIOR :
The libjsig.so library should pass through the sigaction(0x2B) from its interceptor to the OS function.

In fact, the JVM crashes.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
This is the gdb stack trace:

(gdb) whe
#0  0x0000003d0960ce74 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003d09608874 in _L_lock_106 () from /lib64/libpthread.so.0
#2  0x0000003d096082e0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002b30d520ea44 in signal_lock () from /home/murrap/jdk1.7.0_21/jre/lib/amd64/libjsig.so
#4  0x00002b30d520ee40 in sigaction () from /home/murrap/jdk1.7.0_21/jre/lib/amd64/libjsig.so
#5  0x00002b30d5ee606e in VMError::reset_signal_handlers ()
   from /home/murrap/jdk1.7.0_21/jre/lib/amd64/server/libjvm.so
#6  0x00002b30d5ee5b46 in VMError::report_and_die () from /home/murrap/jdk1.7.0_21/jre/lib/amd64/server/libjvm.so
#7  0x00002b30d5d89370 in JVM_handle_linux_signal () from /home/murrap/jdk1.7.0_21/jre/lib/amd64/server/libjvm.so
#8  <signal handler called>
#9  0x00002b30d520eed7 in sigaction () from /home/murrap/jdk1.7.0_21/jre/lib/amd64/libjsig.so
#10 0x00002aaab82eb6ee in Java_TestJNI_doSomething () from /home/murrap/jni/libTestJNI.so

The thread hangs in sigaction() while attempting to acquire a lock that is already held at frame #9 by the application native code.


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
This is the native code. You'd need to construct a caller in java:

#include <stdio.h>
#include <jni.h>
#define __USE_GNU
#include <signal.h>
#include <sys/ucontext.h>

void sig_handler(int sig, siginfo_t *info, ucontext_t *context) {
        int thrNum;

        printf( " HANDLER (1)
 " );
        // Move forward RIP to skip failing instruction
        context->uc_mcontext.gregs[REG_RIP] += 6;
}

JNIEXPORT void JNICALL Java_TestJNI_doSomething(JNIEnv *env, jclass klass, jint val) {
        struct sigaction act;
        struct sigaction oact;
        pthread_attr_t attr;
        stack_t stack;

        act.sa_flags = SA_ONSTACK|SA_RESTART|SA_SIGINFO;
        sigfillset(&act.sa_mask);
        act.sa_handler = SIG_DFL;
        act.sa_sigaction = (void (*)())sig_handler;
        sigaction(0x20+SIGSEGV, &act, &oact);

        printf( " doSomething(%d)
 " , val);
        printf( " old handler = %p
 " , oact.sa_handler);
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
There is no solution for this problem except a modification of the behaviour of jsig.c.