JDK-6545701 : Java plugin triggers Data Execution Prevention (DEP) in 32-bit IE7 on 64-bit Vista
  • Type: Bug
  • Component: deploy
  • Sub-Component: plugin
  • Affected Version: 6
  • Priority: P2
  • Status: Closed
  • Resolution: Duplicate
  • OS: windows_vista
  • CPU: x86
  • Submitted: 2007-04-13
  • Updated: 2010-08-07
  • Resolved: 2008-12-23
Related Reports
Duplicate :  
Description
FULL PRODUCT VERSION :


ADDITIONAL OS VERSION INFORMATION :
Vista Ultimate x64

A DESCRIPTION OF THE PROBLEM :
The Java plug-in for 32-bit Internet Explorer 7 triggers a Data Execution Prevention (DEP) event on 64-bit Vista.  This happens only if DEP is enabled for IE7, which is not the default setting.  

This was observed with a couple of different Java applets, such as Secunia's "Software Inspector" at  http://www.secunia.com/software_inspector

Here are three more links that causes Internet Explorer 7 to shut down with a Data Execution Prevention error on my system under the circumstances I described:

http://www.time.gov and click on a time zone to launch the Java applet there.

http://www.java.com/en/games/desktop/radicalaces.jsp  (random Java game I selected, click "TRY IT NOW!")

http://www.java.com/en/desktop/ultramixer.jsp  (random Java app I selected, click "TRY IT NOW!")



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
A)  enable IE7's dedicated Data Execution Prevention setting by

1)  running IE7 as an Administrator (Start > All Programs > right-click Internet Explorer and choose Run As Administrator)

2)  going to Tools > Internet Options, scrolling down to SECURITY, and enabling memory protection


B)  visit a web site that uses a Java applet, e.g. http://www.secunia.com/software_inspector

or the above links.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
I was expecting the Java plug-in to work normally.
ACTUAL -
Internet Explorer 7 gets shut down with a Data Execution Prevention alert.

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Disabling IE7's Data Execution Prevention setting is a viable workaround.

Comments
EVALUATION this is fixed in 6u12 by fix to: 6674383 : Problems running applets in the Hyper Text Application (HTA) from windows
23-12-2008

WORK AROUND Don't use: bcdedit.exe on Windows Vista: /set nx AlwaysOn /noexecute=alwayson in boot.ini file on Windows XP Please use the following instead: bcdedit.exe on Windows Vista: /set nx OptOut /noexecute=optout in boot.ini file on Windows XP in both cases with empty exclusion list.
07-03-2008