JDK-8164410 : JRE 6u121 causes applet to fail with: Reset deny session certificate store
  • Type: Bug
  • Component: deploy
  • Sub-Component: plugin
  • Affected Version: 7u121
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • Submitted: 2016-08-18
  • Updated: 2018-07-06
  • Resolved: 2016-10-20
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7 JDK 8
7u121Fixed 8u141Fixed
Related Reports
Relates :  
Description
Submitter has upgraded to JDK 6u121 and is seeing TLS connections fail for their applets. Following stack trace is present : 

java.lang.AbstractMethodError: com.sun.net.ssl.internal.ssl.X509ExtendedTrustManager.checkServerTrusted([Ljava/security/cert/X509Certificate;Ljava/lang/String;Ljava/net/Socket;)V
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(Unknown Source)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(Unknown Source)
        at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Unknown Source)
        at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
        at com.sun.deploy.net.HttpUtils.followRedirects(Unknown Source)
        at com.sun.deploy.net.BasicHttpRequest.doRequest(Unknown Source)
        at com.sun.deploy.net.BasicHttpRequest.doGetRequestEX(Unknown Source)

Comments
one alternative approach here might be to rename the X509ExtendedDeployTrustManager deploy class in JDK 6. That'll mean that the JDK 7+ variants of the class won't get loaded. (root cause here) Propose to rename X509ExtendedDeployTrustManager to X509ExtendedDeploy6TrustManager - I'll run some build and tests to see if this could work. It'll mean that JDK 8, 9 edits would be unnecessary.
16-01-2017

workaround : Having only one JRE installed should mean that this issue is not encountered. It only appears if end user has 6u121 or later installed in parallel with a JDK 7u or JDK 8u JRE. If JRE 6 is not required, it could be uninstalled.
22-12-2016

Crucible: https://java.se.oracle.com/code/cru/CR-JDK7UCPU-832
06-10-2016

Turns out that the deploy stack will always use the deploy.jar file from the latest installed JRE on the system. I'm in contact with deploy team about a possible solution. They might be taking ownership of this issue.
31-08-2016

The only class in deploy stack that should implement the com.sun.net.ssl.internal.ssl.X509ExtendedTrustManager.checkServerTrusted(X509Certificate[], String, Socket) method is : com/sun/deploy/security/X509ExtendedDeployTrustManagerDelegate.class I've confirmed that the deploy code was updated to take account of the new TLSv1.2 code changes. The X509ExtendedDeployTrustManagerDelegate class is located in the deploy.jar file. I see that plugin uses the deploy.jar code from JDK 7u. : <-Djava.class.path=C:\Program Files (x86)\Java\jre7\lib\deploy.jar> I need to check with deploy team on why that is the case.
30-08-2016

Strange failure. Not sure if it's config related yet. X509ExtendedTrustManager.checkServerTrusted signature was changed in 6u121 for TLSv1.2 purposes. It's as if the Cu is running with an older class. com/sun/net/ssl/internal/ssl/X509ExtendedTrustManager.class is package in the jsse.jar file. Can you get output of all classes in that jar file from submitterplease. Maybe the install/upgrade didn't go well. 2nd step would be to clear all deploy cache and re-test. If these steps don't work, can you have them switch on verbose class loading and return the output from an applet test. I think -verbose:class will need to be inserted into the JCP runtime parameters.
19-08-2016