FULL PRODUCT VERSION :
jdk1.8.0_74
ADDITIONAL OS VERSION INFORMATION :
Windows 2008 R2 SP1 Enterprise Edition
A DESCRIPTION OF THE PROBLEM :
We have been successfully using PKCS11 provider for encryption of our Tomcat based application. It works fine for version jdk1.8.0_66, but use of PKCS11 (FIPs encryption) fails with higher version of Java 8 updates (including udpate 71, 73 and 74).
When we set following in java.security file:
# List of providers and their preference orders (see above):
security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/pkcs11.cfg
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
security.provider.4=sun.security.ec.SunEC
security.provider.5=com.sun.net.ssl.internal.ssl.Provider
security.provider.6=com.sun.crypto.provider.SunJCE
....................
Our secure connections fails and running Tomcat SSL Debug logs captures following information:
http-bio-443-exec-1, READ: TLSv1 Handshake, length = 262
http-bio-443-exec-1, handling exception: javax.net.ssl.SSLProtocolException: Unable to process PreMasterSecret, may be too big
%% Invalidated: [Session-11, TLS_RSA_WITH_AES_128_CBC_SHA]
http-bio-443-exec-2, SEND TLSv1 ALERT: fatal, description = unexpected_message
http-bio-443-exec-2, WRITE: TLSv1 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 01 00 02 02 0A .......
http-bio-443-exec-2, called closeSocket()
http-bio-443-exec-2, IOException in getSession(): javax.net.ssl.SSLProtocolException: Unable to process PreMasterSecret, may be too big
http-bio-443-exec-2, called close()
http-bio-443-exec-2, called closeInternal(true)
http-bio-443-exec-1, SEND TLSv1 ALERT: fatal, description = unexpected_message
http-bio-443-exec-1, WRITE: TLSv1 Alert, length = 2
--------------------------------------------------------------------------------------------
When we disable pointer to PKCS11.cfg file everything works fine with SSL connections.
REGRESSION. Last worked in version 8u66
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
We enabled following PKCS11 encryption in java.security file to produce the behavior
security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/pkcs11.cfg
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Our SSL connection to our Tomcat based application should successfully complete secure handshake/authentication and application should load.
ACTUAL -
On browser we just see following error:
"SSL connection error ERR_SSL_PROTOCOL_ERROR"
ERROR MESSAGES/STACK TRACES THAT OCCUR :
In Tomcat SSLDebug Logs, we see following captured:
--------------------------------------------------------------------------------------------
http-bio-443-exec-1, READ: TLSv1 Handshake, length = 262
http-bio-443-exec-1, handling exception: javax.net.ssl.SSLProtocolException: Unable to process PreMasterSecret, may be too big
%% Invalidated: [Session-11, TLS_RSA_WITH_AES_128_CBC_SHA]
http-bio-443-exec-2, SEND TLSv1 ALERT: fatal, description = unexpected_message
http-bio-443-exec-2, WRITE: TLSv1 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 01 00 02 02 0A .......
http-bio-443-exec-2, called closeSocket()
http-bio-443-exec-2, IOException in getSession(): javax.net.ssl.SSLProtocolException: Unable to process PreMasterSecret, may be too big
http-bio-443-exec-2, called close()
http-bio-443-exec-2, called closeInternal(true)
http-bio-443-exec-1, SEND TLSv1 ALERT: fatal, description = unexpected_message
http-bio-443-exec-1, WRITE: TLSv1 Alert, length = 2
--------------------------------------------------------------------------------------------
REPRODUCIBILITY :
This bug can be reproduced always.
CUSTOMER SUBMITTED WORKAROUND :
The only way we can use latest Java is by disabling use of PKCS11.cfg file in java.security. With existing security in place, we cannot disable PKCS11.cfg file.
Our temporary workaround is to use 1.8.0_66