JDK-8207029 : Unable to use custom SSLEngine with default TrustManagerFactory after updating to JDK 11 b21
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 11
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2018-07-11
  • Updated: 2020-11-23
  • Resolved: 2018-07-13
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 8 Other
11 b23Fixed 8u261Fixed openjdk8u272Fixed
Related Reports
Relates :  
Relates :  
Relates :  
Description
ADDITIONAL SYSTEM INFORMATION :
# java -version
java version "11-ea" 2018-09-25
Java(TM) SE Runtime Environment 18.9 (build 11-ea+21)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11-ea+21, mixed mode)


A DESCRIPTION OF THE PROBLEM :
Hi all,

I just tried to run netty[1] testsuite with the latest jdk11 EA release (21) and saw some class-cast-exception with our custom SSLEngine implementation:


Caused by: java.lang.ClassCastException: class io.netty.handler.ssl.OpenSslEngine cannot be cast to class sun.security.ssl.SSLEngineImpl (io.netty.handler.ssl.OpenSslEngine is in unnamed module of loader 'app'; sun.security.ssl.SSLEngineImpl is in module java.base of loader 'bootstrap')
	at java.base/sun.security.ssl.SSLAlgorithmConstraints.<init>(SSLAlgorithmConstraints.java:93)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:270)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
	at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:237)
	at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:621)
	... 27 more


This change seems to be related to:
http://hg.openjdk.java.net/jdk/jdk11/rev/68fa3d4026ea

I think you miss an instanceof check here in SSLAlgorithmConstraints before try to cast to SSLEngineImpl, as otherwise it will be impossible to use custom implementations of SSLEngine (which we have in netty) with the default TrustManagerFactory.

REGRESSION : Last worked in version 11

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Create custom SSLEngine implementation and invoke sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(...) with it as argument. 

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Should not throw ClassCastException, X509TrustManagerImpl should not depend on the fact that SSLEngineImpl is used.
ACTUAL -
Throws an ClassCastException like this:

Caused by: java.lang.ClassCastException: class io.netty.handler.ssl.OpenSslEngine cannot be cast to class sun.security.ssl.SSLEngineImpl (io.netty.handler.ssl.OpenSslEngine is in unnamed module of loader 'app'; sun.security.ssl.SSLEngineImpl is in module java.base of loader 'bootstrap')
	at java.base/sun.security.ssl.SSLAlgorithmConstraints.<init>(SSLAlgorithmConstraints.java:93)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:270)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
	at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:237)
	at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:621)
	... 27 more


CUSTOMER SUBMITTED WORKAROUND :
There is no workaround, like stated it worked in ea19 last. That said the fix should be as simple as adding an instanceof SSLEngineImpl check before trying to cast.

FREQUENCY : always



Comments
The following lines may also need instanceof checking before type casting. http://hg.openjdk.java.net/jdk/jdk11/file/2282560a3d29/src/java.base/share/classes/sun/security/ssl/SSLAlgorithmConstraints.java#l128 http://hg.openjdk.java.net/jdk/jdk11/file/2282560a3d29/src/java.base/share/classes/sun/security/ssl/SSLAlgorithmConstraints.java#l149
24-07-2018

Run test suite in Netty sub project handler [1] with JDK 11 build 23, and found the same failure. For example, please see attached io.netty.handler.ssl.JdkConscryptSslEngineInteropTest.txt. In addition, all the same tests passed with JDK 11 build 9. [1] https://github.com/netty/netty/tree/4.1/handler
24-07-2018

Norman Maurer <norman.maurer@googlemail.com>: ------------------------------------------------------------------ "I just noticed you will also need to do the instanceof before you cast to SocketImpl in this class."
11-07-2018

http://mail.openjdk.java.net/pipermail/security-dev/2018-July/017536.html
11-07-2018