JDK-8005607 : Recursion in J2DXErrHandler() Causes a Stack Overflow on Linux
  • Type: Bug
  • Component: client-libs
  • Sub-Component: java.awt
  • Affected Version: 6
  • Priority: P2
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux,solaris
  • CPU: x86
  • Submitted: 2012-12-29
  • Updated: 2014-02-12
  • Resolved: 2013-07-25
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 6 JDK 7 JDK 8
6u51Fixed 7u60Fixed 8Fixed
Related Reports
Relates :  
JDK-8025775 - JNI warnings in TryXShmAttach
Relates :  
JDK-8015730 - PIT: On Linux, OGL=true and fbobject=false leads to deadlock or infinite loop
Description
FULL PRODUCT VERSION :
Any update of 32-bit JDK 6, JDK 7, JDK 8

ADDITIONAL OS VERSION INFORMATION :
32-bit Linux OS, 32-bit Oracle Solaris OS (for example Oracle Solaris 10 8/11 s10x_u10wos_17b X86)

A DESCRIPTION OF THE PROBLEM :
Infinite recursion in AWT native function J2DXErrHandler() causes a stack overflow and a crash under certain circumstances on 32-bit Linux OS or 32-bit Oracle Solaris OS. An approximation of J2DXErrHandler() function in pseudo-code is provided below.

J2DXErrHandler(parms) { 
    if (errorBelongsToJvm(parms)) 
        letJvmHandleIt(parms); 
    else if (pOldHandler) 
        pOldHandler(parms); 
} 
 
The problem occurs, if another thread (for example, GTK thread) is doing the same sort of thing concurrently. This can lead to the following situation.

JVM thread: Sets J2DXErrHandler(), saves ANY_PREVIOUS_HANDLER as previous
    GTK thread: Sets some GTK_HANDLER, saves J2DXErrHandler() as previous
JVM thread: Restores ANY_PREVIOUS_HANDLER
    GTK thread: Restores J2DXErrHandler()
JVM thread: Sets J2DXErrHandler(), saves J2DXErrHandler() as previous

Then, if J2DXErrHandler() is called for an error that JVM doesn't own, this error handler calls a previous handler, which was saved, and passes the error for processing. Unfortunately the previous handler is also set to J2DXErrHandler(), thus the result is an infinite recursion, a stack overflow, and a crash.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Make sure that OS used for this purpose is 32-bit Linux OS or 32-bit Oracle Solaris OS.
2. Download the attached "J2DXErrHandlerTestcase.zip" or "J2DXErrHandlerTestcase-Solaris.zip" archive depending on the OS type and unpack it.
3. Execute: java -cp xtest.jar -Djava.library.path=. xtest
    or unpack "src.zip", compile the test case using "makefile" script and execute the previously mentioned command line.
4. When Java2Demo window appears select the "Images" tab.
5. Select the menu item "Options->Background Color".
6. After the window appears, hit "Escape" to close it.
7. Repeat steps 5 and 6 in quick succession until the process crashes (may take 10-20 attempts).

Notes: 
- The problem can be reproduced using tabs other than "Images".
- The problem may also be easier to reproduce on multi-core hosts.

REPRODUCIBILITY :
The problem is intermittent and may be reproduced using the test case after different number of attempts.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
$ /shared/jdks/oracle/6/jdk6_37/bin/java -cp xtest.jar -Djava.library.path=. xtest 
Loading library
Starting xtest
Starting xtest: xesetTime=10 xeclearTime=10
Opening X display
Starting thread...
Waiting for thread to run...
Thread started
Thread running
Launch java2 demo 
ODD 0x9fd9f950 != 0xb6b4b902
ODD 0x9f45b280 != 0xb6b4b902 
ODD 0x9f465c40 != 0xb6b4b902
Launch complete
ODD 0x9f465c40 != 0xb6b4b902
URK! newXeh called 
URK! newXeh called
URK! newXeh called 
URK! newXeh called
ODD 0x9f465c40 != 0xb6b4b902 
ODD 0x9f465c40 != 0xb6b4b902
Segmentation fault
Comments
Reusing the bug#. As 6u72 merged into 6u71 and already have 6u71, using this bug for 6u65-b31. http://closedjdk.us.oracle.com/jdk6u/jdk6u65-psu/j2se/rev/6cf06b255512
19-09-2013
Added 6bpr-c-watch given CU demand.
26-07-2013
Test case compatible with Oracle Solaris OS was attached.
26-04-2013
It was found out that this bug is also present on 32-bit Oracle Solaris OS. The bug was reproduced on OS of the exact version "Oracle Solaris 10 8/11 s10x_u10wos_17b X86" with the following releases of JDK: - JDK 6u45 b06 - JDK 7u21 b11 - JDK 8 b86 The fields "Description" and "OS" of this issue were edited to reflect this new detail.
26-04-2013
The reason of a crash of any Swing application on JDK with a fix, which is based on substitution of J2DXErrHandler() native handler for a similar synthetic Java handler called by AWT native global error handler, is initialization of sun.awt.X11.XToolkit class after a call to its static WITH_XERROR_HANDLER method from native code in "src/solaris/native/sun/awt/awt_GraphicsEnv.c", "src/solaris/native/sun/java2d/x11/X11SurfaceData.c". It was found out that static initialization block of XToolkit class depends on other singletones like java.awt.GraphicsEnvironment class, whose initialization in its turn becomes dependent on XToolkit class's static methods after the fix is applied. Such situation creates cycle in the process of initialization of GraphicsEnvironment singleton, for which the existing code is not prepared. To eliminate this problem all XError handling code can be moved from XToolkit class to a separate class, which does not depend on any renowned singletone objects.
30-01-2013
There is the already fixed bug JDK-6678385 which is about a similar problem conceptually. A solution for that bug was introduction of one global error handler for the AWT library on a native level and many synthetic error handlers on Java level. One of possible solutions for the current bug could be substitution of J2DXErrHandler() native handler for a similar synthetic handler in Java code, which would be called by AWT native global error handler. It was done in local test environment by change of EXEC_WITH_XERROR_HANDLER(J2DXErrHandler, ... for a pair of calls: sun.awt.X11.XTookit.WITH_XERROR_HANDLER(SynthJ2DXErrHandler) sun.awt.X11.XTookit.RESTORE_XERROR_HANDLER() in "src/solaris/native/sun/awt/awt_GraphicsEnv.c", "src/solaris/native/sun/java2d/x11/X11SurfaceData.c" files of JDK. Though all underlying JNI calls were implemented and JDK was compiled successfully, execution of any Swing application lead to a crash in loading of Fonts, and that crash could not be resolved. Therefore another solution requiring less source code changes was designed. The root cause of such a bug is the fact that any process thread which called XSetErrorHandler native function restores a previous error handler without verification that the handler set by this thread is still current for the process. Since such threads can be created by third-party libraries, a mechanism of setting and restoring the error handler is out of control of JDK. And if code restoring a previous error handler only, when a current handler is that one, which was set by a current thread, is implemented in JDK, there is no way to control such behavior in third-party libraries. The designed solution is based on introduction of the logic, which detects indirect recursive calls to J2DXErrHandler by means of a simple counter, to J2DXErrHandler native function. Such a solution requires minimum code changes, does not alter the handler's code significantly and eliminates this bug.
29-12-2012