Bug ID: JDK-8025775 JNI warnings in TryXShmAttach

Type: Bug
Component: client-libs
Sub-Component: java.awt
Affected Version: 7u40,8

Priority: P2
Status: Closed
Resolution: Fixed
OS: linux
CPU: generic

Submitted: 2013-10-01
Updated: 2015-04-29
Resolved: 2013-12-04

JDK 6	JDK 7	JDK 8
6u101Fixed	7u60Fixed	8 b120Fixed

Run netbeans 7.3.1 with -Xcheck:jni. With JDK 7u40, only a few, seemingly harmless warnings are printed. With JDK8 b106, a large number of warnings are printed:

Warning: Calling other JNI functions in the scope of Get/ReleasePrimitiveArrayCritical or Get/ReleaseStringCritical
Warning: Calling other JNI functions in the scope of Get/ReleasePrimitiveArrayCritical or Get/ReleaseStringCritical
Warning: Calling other JNI functions in the scope of Get/ReleasePrimitiveArrayCritical or Get/ReleaseStringCritical
Warning: Calling other JNI functions in the scope of Get/ReleasePrimitiveArrayCritical or Get/ReleaseStringCritical
...

The JNI functions that cause these warnings are Java_sun_java2d_loops_Blit_Blit and Java_sun_java2d_loops_MaskBlit_MaskBlit. They are trying to copy a source image (in memory buffer) to a destination image (in X11).

[1] It first calls BufImg_GetRasInfo, which calls jni_GetPrimitiveArrayCritical,  which calls GC_locker::jni_lock to prevent GC from happening.

[2] Then, it calls X11SD_GetRasInfo, which tries to execute a Java method:

    #0 functionEnter (thr=0x7f69e4170800) at src/share/vm/prims/jniCheck.cpp:187
    #1 0x00007f6a6b987b3c in checked_jni_EnsureLocalCapacity (env=0x7f69e4170a20, capacity=3)
    #2 0x00007f6a6aa934ac in JNU_CallStaticMethodByName ()
    #3 0x00007f6a44d4113f in TryXShmAttach ()
    #4 0x00007f6a44d32825 in X11SD_CreateSharedImage ()
    #5 0x00007f6a44d33eb4 in X11SD_GetImage ()
    #6 0x00007f6a44d3464a in X11SD_GetRasInfo ()
    #7 0x00007f6a45062ee1 in Java_sun_java2d_loops_Blit_Blit ()

Actually a few different methods are called while the primitive array is still being held.

If one of these Java methods causes a GC, we will deadlock because GC has been disabled. 

The chance for deadlock may be higher for apps which refreshes the screen frequently and creates a lot of garbage.

Taking into account this is a customer escaltion and jdk6 only critical request SQE OK to take it in CPU15_03
29-04-2015
Verified in JDK 8b120 No JNI warnings on Netbeans 7.3.1 startup with 8b120
23-12-2013
SQE is ok to take the fix in 7u60 as it resolves regression introduced in 7u60 b01.
12-12-2013
Anton Litvinov added a comment - 2013-12-09 08:17 - Restricted to Confidential - edited 7u60-critical-request-dev justification: It is a backport of the fix from JDK 8 to JDK 7, which eliminates a possible deadlock being a regression introduced in JDK 7u60 b01. - Webrev: http://cr.openjdk.java.net/~alitvinov/8025775/jdk7/webrev.00 - Review thread: Approval 1 - http://mail.openjdk.java.net/pipermail/awt-dev/2013-December/006567.html Approval 2 - http://mail.openjdk.java.net/pipermail/awt-dev/2013-December/006568.html - JDK 8 changeset: http://hg.openjdk.java.net/jdk8/awt/jdk/rev/233cc95e1a0a - Impact: 2D, MToolkit code areas of JDK on Linux OS, Solaris OS.
12-12-2013
Release team: Approved for fixing
03-12-2013
SQE: OK to push in jdk8
03-12-2013
A new solution was created and reviewed by the code reviewers at awt-dev@openjdk.java.net e-mail alias. The solution is based on: - Reversion of all XError handling code implemented as "native -> Java" calls in the fix for the bug JDK-8005607 to its prior state with native error handlers. - Introduction of a native synthetic error handler "awt_util.c::current_native_xerror_handler" set/unset through previously known macros in "awt_util.h" file and called from "XlibWrapper.c::ToolkitErrorHandler" function before a call to Java synthetic error handler.
02-12-2013
Although the solution #2 would resolve JNI warnings, it would also be unacceptable, because it would not let the code in "TryXShmAttach" function to define whether XError was or was not generated during the call to "XShmAttach" function without querying Java synthetic error handler's field "errorOccurred" through "native -> Java" call.
02-12-2013
Additional research showed that calling "XShmAttach" function does lead to generation of XError. An example message of XError event generated by "XShmAttach" function in the local test environment is shown below: X Error of failed request: BadAccess (attempt to access private resource denied) Major opcode of failed request: 130 (MIT-SHM) Minor opcode of failed request: 1 (X_ShmAttach) Serial number of failed request: 18 Current serial number in output stream: 21 It was practically proven that "XShmAttach" function generates XError events, when the test case is executed on a remote DISPLAY through SSH protocol. Also it was proven that, when such XError is generated, "XShmAttach" function still returns "1" value indicating success instead of "0" value indicating failure. Therefore return value of "Bool" type of the function "XShmAttach" is unreliable, what means that, unfortunately, removal of X error handling code from JDK's native "TryXShmAttach" function is unacceptable solution for this bug.
29-11-2013
Solution #2 above could still be applicable: we could install an error handler before critically locking an array, and uninstall it after the array is released. This way no JNI calls would be performed while the array is being locked.
29-11-2013
Reproduced JNI warnings specified in the bug's description by running NetBeans IDE 7.3.1 with JDK 8, JDK 7 compiled from the latest sources in jdk8/awt, jdk7u-dev repositories. Verified that all warnings with the message "Warning: Calling other JNI functions in the scope of Get/ReleasePrimitiveArrayCritical or Get/ReleaseStringCritical" are results of Java methods Invocations in the native function "TryXShmAttach" added into the file "jdk/src/solaris/native/sun/awt/awt_GraphicsEnv.c" by the fix for JDK-8005607. Removal of Java methods Invocations in the function "TryXShmAttach" eliminates all these JNI warnings both in JDK 8 and JDK 7. Reason of fewer number of JNI warnings displayed with JDK 7u40 is explained by absence of the fix for JDK-8005607 (integrated into JDK 7u60 b01) in JDK 7u40. As a possible solution I consider the solution suggested by Anthony Petrov, which consists in omission of installation, removal and querying of the synthetic "XShmAttachHandler" X error handler in "TryXShmAttach" function, because: - The specification of "XShmAttach" native function states that, if the function call fails, "0" value is returned. This function call result may be used to determine whether the function succeeded or not. - In case, when some X error is generated during the call to "XShmAttach" function Java application should not crash, since XToolkit global error handler is already set. As soon as the solution is ready, all known 2D/AWT/Swing regression tests will be executed on Linux OS and Solaris OS.
28-11-2013
Release team: Approved for deferral. Putting this on 8-pool so you can decide which 8u release to fix this in.
18-10-2013
I think that potential performance regressions could be significant if we choose to avoid critical array access. Regardless, there's two more options: 1. Evaluate whether we really have to install an X error handler when calling the XShmAttach() function. Its specification suggests that this function can't generate an error. 2. If #1 is untrue, then we could consider installing the XError handler somewhere earlier in the code path (and uninstalling it later, correspondingly), so that all critical array access is happening while the handler is installed. This solution is guaranteed to fix the issue.
16-10-2013
I would ask Anton to do some more investigation. The easiest fix is to use Get*ArrayElements() instead of GetPrimitiveArrayCritical(), but it can cause some performance regression. If the regression is acceptable (i.e. if this native method is rarely used), it would be fine to include this fix to 8.
16-10-2013
I don't understand why this bug is considered "a potential deadlock". There is no way to force GC, so regardless of what Java up-calls are, the deadlock should not happen. I agree that it's a bug, and some bad things can happen, when JNI is used from Get/ReleasePrimitiveArrayCritical(), but I don't see why it's a deadlock.
16-10-2013
jdk8: SQE OK to defer
16-10-2013
There was described a potential, hypothetical deadlock. It could occur if any of the Java up-calls cause a GC to start. Since we use array critical sections, the GC is locked already, and thus it wouldn't be able to start. However, I believe that the likelihood of this event is extremely low because the Java up-calls that we use don't produce a lot of memory allocations to cause starting GC.
16-10-2013
What about deadlock mentioned in Description? I don't see anything about it in these other fixed bugs.
16-10-2013
Pending SQE approval
16-10-2013
The bug JDK-8025568 was unbound from this bug, because they are not related to each other, and a bug against Ubuntu was filed for JDK-8025568.
02-10-2013
This looks like a regression of JDK-8005607, which introduced the TryXShmAttach() function. This function now uses a new utility class XErrorHandlerUtil to install and uninstall a custom X Error Handler. Hence the JNI calls. Assigning this bug to an engineer who developed the original fix for evaluation.
01-10-2013