FULL PRODUCT VERSION :
java version "1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) Client VM (build 25.72-b15, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
OS independent, but tested on Windows 7.

EXTRA RELEVANT SYSTEM CONFIGURATION :
Irrelevant for the issue

A DESCRIPTION OF THE PROBLEM :
This bug has been introduced in the RSAClientKeyExchange with change:
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/rev/3e9d6880f36f

The old (working) code looked like this:
            Cipher cipher = JsseJce.getCipher(JsseJce.CIPHER_RSA_PKCS1);
            cipher.init(...);

The new (broken) code does this:
            Cipher cipher = JsseJce.getCipher(JsseJce.CIPHER_RSA_PKCS1);
            boolean needFailover = !KeyUtil.isOracleJCEProvider(
                    cipher.getProvider().getName());
            if (needFailover) {
                cipher.init(Cipher.DECRYPT_MODE, privateKey);

Even shorter, it used to be:
- cipher.init

And now it's:
- cipher.getProvider()
- cipher.init

The problem with this is that calling cipher.getProvider before cipher.init breaks delayed provider selection. It forces the first provider with a matching algorithm to be selected (ignoring the key type).
This will cause an error when a incompatible key from a different provider then the first is supplied.
For example, this breaks if a PKCS11 provider (with a RSA keystore) is used, which is not the first in the list of providers. In old versions this would work without problem.

The problem manifests itself as failed SSL connections, with the following in the debug log (-Djavax.net.debug=all):
"http-bio-443-exec-6, handling exception: javax.net.ssl.SSLProtocolException: Unable to process PreMasterSecret, may be too big" The error message isn't quite correct, it's the default handling of a InvalidKeyException in RSAClientKeyExchange.




REGRESSION.  Last worked in version 6u45

ADDITIONAL REGRESSION INFORMATION: 
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b15)
Java HotSpot(TM) Client VM (build 25.45-b02, mixed mode)


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
- Use a PKCS11 provider (hardware token).
- Make sure the PKCS11 provider is the last in the list of security providers.
- Start a SSL connection using a RSA key from a keystore from the PKCS11 provider.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
SSL connections is set up without issue.
ACTUAL -
SSL Connection fails, an error is logged in the debug log (-Djavax.net.debug=all):
"http-bio-443-exec-6, handling exception: javax.net.ssl.SSLProtocolException: Unable to process PreMasterSecret, may be too big" The error message isn't quite correct, it's the default handling of a InvalidKeyException in RSAClientKeyExchange.


ERROR MESSAGES/STACK TRACES THAT OCCUR :
"http-bio-443-exec-6, handling exception: javax.net.ssl.SSLProtocolException: Unable to process PreMasterSecret, may be too big" The error message isn't quite correct, it's the default handling of a InvalidKeyException in RSAClientKeyExchange.


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Since a pkcs11 provider/hardware token is required, I can't provide a self contained sample. The description should be quite clear though.

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
It might possibly work if you add the PKCS11 provider as the first provider in the list of security providers. But this is not always a acceptable/possible workaround.