JDK-6587371 : JCK SystemTray tests cause VM crash on WinVista in JDK 6u3 with -Xcomp option (passed on 6u1)
  • Type: Bug
  • Component: client-libs
  • Sub-Component: java.awt
  • Affected Version: 6.0a,6u3
  • Priority: P2
  • Status: Resolved
  • Resolution: Fixed
  • OS: windows_xp,windows_vista
  • CPU: x86
  • Submitted: 2007-07-31
  • Updated: 2012-03-22
  • Resolved: 2007-09-12
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
6u4 b03Fixed
Related Reports
Duplicate :  
Relates :  
OS/Platform: Windows Vista x32 ( also it can be reproduced on WinVista x64 with x32 JDK )
VM opts: -Xcomp
  failed on: 6u3 b01,  6u2( all builds ), jdk6u2 ER Fiducia - windows-i586
  passed on: 6u1
  also passed: without -Xcomp opt (all updates/builds), on windows-amd64 (all upd/bld)
Failure type: VM crash
  JCK 6a b14
How to reproduce:
See comments to get access on test Vista hosts:
On stt-27.russia or stt-40.russia under MKS bash:
export  JAVA_HOME=Z:/Links/stt/jdk_promotions/JDK6u3/b01/binaries/windows-i586/jdk1.6.0_03
export  JAVA_COMMON_OPTS="-client -Xcomp -Xfuture"
export  CP="Z:/Links/stt/jck_promotions/6a/fcs/b14/binaries/JCK-runtime-6a/classes"
export  JAVA_TEST="javasoft.sqe.tests.api.java.awt.SystemTray.SystemTrayMiscTests"

$JAVA_HOME/bin/java.exe $JAVA_COMMON_OPTS -classpath $CP $JAVA_TEST
or use Z:\jck_workspace\manual_run\dg159268_runs\failed_tests\unsorted\jdk6u3_Vista\rerun.sh
Z is mapped as \\stt-13\export\stt

Additional info:
   hs_* files are attached ( hs_err_pid1872.log - common JDK 6u3 bundle,
                             hs_err_pid5636.log - info from fastdebug bundle )
Test output sample:
command: com.sun.jck.lib.ExecJCKTestOtherJVMCmd PATH=Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\lib\\win32;Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\lib\\win32\\jmx;Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\lib\\win32 SystemRoot=C:/Windows TMP=Z:\\Links\\stt\\jck_workspace\\jdk6u2_ER_Fiducia\\jck\\runtime\\WinVista-x86-JCK6.0-runtime\\workDir TEMP=Z:\\Links\\stt\\jck_workspace\\jdk6u2_ER_Fiducia\\jck\\runtime\\WinVista-x86-JCK6.0-runtime\\workDir C:\\jdk\\ER_6_02_Fiducia\\jre1.6.0_02\\bin\\java.exe -client -Xcomp -Djava.ext.dirs=Z:/Links/stt/jck_promotions/6a/fcs/b14/binaries/JCK-runtime-6a/lib/extensions;C:/jdk/ER_6_02_Fiducia/jre1.6.0_02//lib/ext;C:/jdk/ER_6_02_Fiducia/jre1.6.0_02//jre/lib/ext -Xfuture -classpath Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\classes;C:\\jdk\\ER_6_02_Fiducia\\jre1.6.0_02\\lib\\tools.jar -Djava.security.policy=Z:\\Links\\stt\\jck_promotions\\6a\\fcs\\b14\\binaries\\JCK-runtime-6a\\lib\\jck.policy javasoft.sqe.tests.api.java.awt.TrayIcon.MouseMotionListenersTests
# An unexpected error has been detected by Java Runtime Environment:
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d92e426, pid=3620, tid=468
# Java VM: Java HotSpot(TM) Client VM (1.6.0_02-erdist-b20072507 compiled mode)
# Problematic frame:
# V  [jvm.dll+0x16e426]
# An error report file with more information is saved as hs_err_pid3620.log
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
result: Failed. unexpected exit code: exit code 1
test result: Failed. unexpected exit code: exit code 1

hs_err* :
# An unexpected error has been detected by Java Runtime Environment:
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d9e4376, pid=1872, tid=832
# Java VM: Java HotSpot(TM) Client VM (1.6.0_03-ea-b01 compiled mode windows-x86)
# Problematic frame:
# V  [jvm.dll+0x174376]
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp

---------------  T H R E A D  ---------------

Current thread (0x0ca5b000):  JavaThread "AWT-EventQueue-0" [_thread_in_vm, id=832, stack(0x0d3e0000,0x0d430000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000000

EAX=0x00000000, EBX=0x0c9f5168, ECX=0x0d42f308, EDX=0x00000000
ESP=0x0d42f2a4, EBP=0x0caff038, ESI=0xffffffff, EDI=0x028000c8
EIP=0x6d9e4376, EFLAGS=0x00010246

Top of Stack: (sp=0x0d42f2a4)
0x0d42f2a4:   028000c8 0ca5b000 0d42f434 00000001
0x0d42f2b4:   0ca5b000 0ca5b000 00000000 00000000
0x0d42f2c4:   012142f8 0c9f5160 0c9f5168 0c9f5554
0x0d42f2d4:   00560000 00560148 0d42f328 0ca5b000
0x0d42f2e4:   6d8c69a5 027fff08 028000c8 0d42f308
0x0d42f2f4:   00000000 00000000 0ca5b000 0d42f558
0x0d42f304:   02800455 00000000 00000000 0ca5b000
0x0d42f314:   c10028e9 0000000a 770c13dd 0058c8a0 

Instructions: (pc=0x6d9e4376)
0x6d9e4366:   89 44 24 18 8d 9b 00 00 00 00 8b 4c 24 4c 8b 11
0x6d9e4376:   8b 02 8b 70 04 85 f6 c6 44 24 50 00 75 04 33 c9 

Stack: [0x0d3e0000,0x0d430000],  sp=0x0d42f2a4,  free space=316k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x174376]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  sun.awt.windows.WToolkit.isProtectedMode()Z
C  0x0d42f478

[error occurred during error reporting (printing Java stack), id 0xc0000005]

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
=>0x0ca5b000 JavaThread "AWT-EventQueue-0" [_thread_in_vm, id=832, stack(0x0d3e0000,0x0d430000)]
  0x0ca59800 JavaThread "AWT-Windows" daemon [_thread_in_native, id=2220, stack(0x0d350000,0x0d3a0000)]
  0x0ca59000 JavaThread "AWT-Shutdown" [_thread_blocked, id=12, stack(0x0ce60000,0x0ceb0000)]
  0x0cafe000 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=2116, stack(0x0cdd0000,0x0ce20000)]
  0x0c910400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1460, stack(0x0cb40000,0x0cb90000)]
  0x00e07c00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=1000, stack(0x0c8b0000,0x0c900000)]
  0x00e07400 JavaThread "Attach Listener" daemon [_thread_blocked, id=1052, stack(0x0c820000,0x0c870000)]
  0x00e03400 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2028, stack(0x0c790000,0x0c7e0000)]
  0x00df5000 JavaThread "Finalizer" daemon [_thread_blocked, id=1304, stack(0x0c700000,0x0c750000)]
  0x00df1400 JavaThread "Reference Handler" daemon [_thread_blocked, id=2272, stack(0x0c670000,0x0c6c0000)]
  0x00d71c00 JavaThread "main" [_thread_blocked, id=2988, stack(0x00150000,0x001a0000)]

Other Threads:
  0x00ded000 VMThread [stack: 0x0c620000,0x0c670000] [id=1236]
  0x0c910c00 WatcherThread [stack: 0x0cbd0000,0x0cc20000] [id=1784]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

 def new generation   total 960K, used 430K [0x04620000, 0x04720000, 0x04b00000)
  eden space 896K,  48% used [0x04620000, 0x0468bab0, 0x04700000)
  from space 64K,   0% used [0x04700000, 0x04700000, 0x04710000)
  to   space 64K,   0% used [0x04710000, 0x04710000, 0x04720000)
 tenured generation   total 4096K, used 0K [0x04b00000, 0x04f00000, 0x08620000)
   the space 4096K,   0% used [0x04b00000, 0x04b00000, 0x04b00200, 0x04f00000)
 compacting perm gen  total 12288K, used 4381K [0x08620000, 0x09220000, 0x0c620000)
   the space 12288K,  35% used [0x08620000, 0x08a67758, 0x08a67800, 0x09220000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00423000 	Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\bin\java.exe
0x77690000 - 0x777e0000 	C:\Windows\SysWOW64\ntdll.dll
0x76b40000 - 0x76c50000 	C:\Windows\syswow64\kernel32.dll
0x76e80000 - 0x76f3f000 	C:\Windows\syswow64\ADVAPI32.dll
0x771e0000 - 0x772d0000 	C:\Windows\syswow64\RPCRT4.dll
0x75890000 - 0x758f0000 	C:\Windows\syswow64\Secur32.dll
0x7c340000 - 0x7c396000 	Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\msvcr71.dll
0x6d870000 - 0x6dac0000 	Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\client\jvm.dll
0x76fd0000 - 0x770a0000 	C:\Windows\syswow64\USER32.dll
0x75d90000 - 0x75e20000 	C:\Windows\syswow64\GDI32.dll
0x751f0000 - 0x75223000 	C:\Windows\system32\WINMM.dll
0x770a0000 - 0x7714a000 	C:\Windows\syswow64\msvcrt.dll
0x75c40000 - 0x75d84000 	C:\Windows\syswow64\ole32.dll
0x76f40000 - 0x76fcc000 	C:\Windows\syswow64\OLEAUT32.dll
0x751b0000 - 0x751e8000 	C:\Windows\system32\OLEACC.dll
0x75e20000 - 0x75e80000 	C:\Windows\system32\IMM32.DLL
0x75f20000 - 0x75fe7000 	C:\Windows\syswow64\MSCTF.dll
0x75490000 - 0x754bc000 	C:\Windows\system32\apphelp.dll
0x77670000 - 0x77679000 	C:\Windows\syswow64\LPK.DLL
0x758f0000 - 0x7596d000 	C:\Windows\syswow64\USP10.dll
0x6d320000 - 0x6d328000 	Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\hpi.dll
0x75b00000 - 0x75b07000 	C:\Windows\syswow64\PSAPI.DLL
0x6d820000 - 0x6d82c000 	Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\verify.dll
0x6d3c0000 - 0x6d3df000 	Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\java.dll
0x6d860000 - 0x6d86f000 	Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\zip.dll
0x6d0b0000 - 0x6d1de000 	Z:\Links\stt\jdk_promotions\JDK6u3\b01\binaries\windows-i586\jdk1.6.0_03\jre\bin\awt.dll
0x75160000 - 0x751a1000 	C:\Windows\system32\WINSPOOL.DRV
0x75690000 - 0x75710000 	C:\Windows\system32\uxtheme.dll
0x75be0000 - 0x75c35000 	C:\Windows\syswow64\SHLWAPI.dll
0x752f0000 - 0x75484000 	C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6000.16386_none_5d07289e07e1d100\comctl32.dll

VM Arguments:
jvm_args: -Xcomp -Xverify:all 
java_command: javasoft.sqe.tests.api.java.awt.SystemTray.SystemTrayMiscTests
Launcher Type: SUN_STANDARD

Environment Variables:
PROCESSOR_IDENTIFIER=EM64T Family 15 Model 6 Stepping 4, GenuineIntel

---------------  S Y S T E M  ---------------

OS: Windows Vista Build 6000 

CPU:total 2 (2 cores per cpu, 1 threads per core) family 15 model 6 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3

Memory: 4k page, physical 2078032k(1429580k free), swap 4194303k(3698464k free)

vm_info: Java HotSpot(TM) Client VM (1.6.0_03-ea-b01) for windows-x86, built on Jul  3 2007 01:45:06 by "java_re" with unknown MS VC++:1310

time: Mon Jul 30 22:58:14 2007
elapsed time: 4 seconds


It looks like workaround for 6490142 related issue.
In WToolkit.java in IsTraySupported() we are calling native method IsProtectedMode() from awt_Toolkit.cpp
It causes VM crash when we are using -Xcomp VM option.

EVALUATION There are two places that need to be fixed (awt_Toolkit.cpp file, Java_sun_awt_windows_WToolkit_isProtectedMode function): 1) The Java_sun_awt_windows_WToolkit_isProtectedMode function should be declared as extern "C" according to the JNI header file generated by the javah utility; 2) The exported Win32 API IsProtectedModeProcess function should be explicitly typedef'ed as WINAPI, otherwise __cdecl calling convention is used. This cause to mispatch in calling conventions.

SUGGESTED FIX ------- awt_Toolkit.cpp ------- *** /tmp/sccs.Cha4PQ Fri Aug 3 15:00:25 2007 --- awt_Toolkit.cpp Fri Aug 3 14:59:51 2007 *************** *** 2319,2325 **** * Method: isProtectedMode * Signature: ()Z */ ! JNIEXPORT jboolean JNICALL Java_sun_awt_windows_WToolkit_isProtectedMode(JNIEnv *env, jclass cls) { TRY; --- 2319,2325 ---- * Method: isProtectedMode * Signature: ()Z */ ! extern "C" JNIEXPORT jboolean JNICALL Java_sun_awt_windows_WToolkit_isProtectedMode(JNIEnv *env, jclass cls) { TRY; *************** *** 2331,2337 **** BOOL bIEIsProtectedModeProcess = FALSE; HINSTANCE hLibIEFrameDll = ::LoadLibrary(TEXT("IEFRAME.DLL")); ! typedef HRESULT IEIsProtectedModeProcessFunc(BOOL *); if (hLibIEFrameDll != NULL) { --- 2331,2337 ---- BOOL bIEIsProtectedModeProcess = FALSE; HINSTANCE hLibIEFrameDll = ::LoadLibrary(TEXT("IEFRAME.DLL")); ! typedef HRESULT (WINAPI IEIsProtectedModeProcessFunc)(BOOL *); if (hLibIEFrameDll != NULL) {

EVALUATION isProtectedMode has a typedef for a function pointer but it doesn't specify the proper calling convention. This causes the code generation to assume a calling convention where it is responsible for removing the arguments. 00000000`04cb2969 51 push ecx 00000000`04cb296a ff55e0 call dword ptr [ebp-20h] 00000000`04cb296d 83c404 add esp,4 So it emits the add esp,4, when in fact the callee has already removed the arguments. This changes the stack depth so the code which is restoring the callee saved arguments restores the wrong values. Windows API functions are always __stdcall but the default is __cdecl.

EVALUATION It looks like the code in the native method sun.awt.windows.WToolkit.isProtectedMode is corrupting a callee save register that the native wrappers keep the thread in. When it returns and performs the pending exception check it's testing the wrong piece of memory. It happens that the memory location it reads is non-null so we go to forward_exception which reloads the thread from TLS and tests the real location and complains that there's no pending exception. In product mode we'd crash because we'd try to dispatch a null exception. The wrappers might want to be more resilient and keep the thread in a stack slot. I still haven't figured out where it goes wrong but it looks like the native method is attempting the properly restore the values it dutifully saved at the beginning but the values it is reading back are wrong, possibly because esp is off by one. The reason this only shows up on Vista is that this code only does something meaningful on Vista. It simply returns false on other releases.