JDK-8223940 : Private key not supported by chosen signature algorithm
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 11,12,13
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: windows_7
  • CPU: x86_64
  • Submitted: 2019-05-09
  • Updated: 2020-11-27
  • Resolved: 2019-11-06
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 13 JDK 14 JDK 8 Other
11.0.8-oracleFixed 13.0.6Fixed 14 b23Fixed 8u261Fixed openjdk8u275Fixed
Description
A DESCRIPTION OF THE PROBLEM :
When using a PKCS11 keystore (using the SunPKCS11 Provider) for client authentication in TLSv1.2, the client chooses a signature algorithm which does not support the P11PrivateKey.

The server sends a list of supported SignatureSchemes for use in client authentication. The method sun.security.ssl.SignatureScheme.getPreferableAlgorithm(List<SignatureScheme>, PrivateKey, ProtocolVersion) chooses which SignatureScheme to use in client authentication. It can choose any algorithm for which a provider exists, even if the algorithm's java.security.Provider$Service.supportsParameter(privateKey) == false. In this case it chooses RSASSA-PSS from the provider sun.security.rsa.SunRsaSign.

Because of this, signing with P11PrivateKey fails (with a misleading message stating no providers support the key, when it's a question of the <algorithm name> implementations not supporting the key):

Caused by: javax.net.ssl.SSLHandshakeException: Cannot produce CertificateVerify signature
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
	at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:590)
	at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:740)
	at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
	at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:173)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1356)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1331)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:241)
	<snip>
Caused by: java.security.InvalidKeyException: No installed provider supports this key: sun.security.pkcs11.P11Key$P11PrivateKey
	at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1163)
	at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1204)
	at java.base/java.security.Signature.initSign(Signature.java:546)
	at java.base/sun.security.ssl.SignatureScheme.getSignature(SignatureScheme.java:473)
	at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:580)
	... 26 more

The problem cannot be worked around by adding RSASSA-PSS to the java.security configuration jdk.tls.disabledAlgorithms property.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Regular Java TLS client using PKCS11 KeyStore from the SunPKCS11 provider for client authentication.

TLSv1.2 server with client authentication, with supported_signature_algorithms preferring RSASSA-PSS (not supported by PKCS11 provider) over RSA_PKCS1_SHA256 (supported by PKCS11 provider).

Open connection from client to server.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The client chooses the RSA_PKCS1_SHA256 signature algorithm. The TLS connection is established with PKCS11 for client authentication.
ACTUAL -
The client chooses the RSASSA-PSS signature algorithm. The client cannot complete TLS handshake because it cannot use the P11PrivateKey in the signature algorithm.

CUSTOMER SUBMITTED WORKAROUND :
Configure server so that supported_signature_algorithms prefers signature algorithms supported by the SunPKCS11 provider (RSA_PKCS1_SHA256, RSA_PKCS1_SHA384, RSA_PKCS1_SHA_512, RSA_SHA224, RSA_PKCS1_SHA1).

FREQUENCY : always
Comments
Fix request (13u): applicable to 13u as well and should be downported. Patch applies clean at this moment; all relevant regtests do pass.
27-11-2020
Fix Request [8u]: Fix is needed now that 8u272 has the TLSv1.3 stack from 11u. If a client prefers RSA-PSS, but the provider does not support it, it will fail as above rather than falling back on other providers. Regarding the workaround, I could not see any obvious way for the server side code to just be configured to reject RSA-PSS. In fact, the bug states that signatures can't be added to jdk.tls.disabledAlgorithms. The fix applies cleanly once automated shuffling is applied with the 11u & 9u scripts.
30-10-2020
(11u) RFR is available: http://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-June/003257.html
09-06-2020
[~mbaesken] Given that there is no patch and we are already in ramp down for July and there is a work-around - configure server so that it prefers signature algorithms supported by the SunPKCS11 provider. It seems a corner case. I'm setting jdk11u-critical-no. Please request jdk11u-fix-request instead.
04-06-2020
jdk11 backport request I would like to have the patch in openjdk11 as well (for better parity with 11.0.8_oracle). The patch does not apply cleanly, RFR follows.
03-06-2020
URL: https://hg.openjdk.java.net/jdk/jdk/rev/d6e682e8fcc3 User: xuelei Date: 2019-11-06 17:45:19 +0000
06-11-2019
This bug may have compatibility impact, please consider backport to supported releases.
24-10-2019
The TLS code could fall back to use the next preferred signature algorithm for the specific key. However, it could be effective if the provider could check if the specific algorithms and parameters are supported. The idea to use java.security.Provider$Service.supportsParameter(privateKey) does not actually work as the signature parameters are expected to work with multiple providers and keys. It is not possible to have the signature parameters provider know the key provider details. So SignatureParameter.provider.service.supportsParameter(privateKey) may not work as expected. Maybe, privateKey.provider.service.supportsParameter(RSASSA-PSSParamter) could work. However, the current provider does not support to check signature algorithm parameters. It may be a potential improvement.
22-10-2019
Re-assigning this to Xuelei so he can evaluate if TLS code can detect this issue and switch to a different signature impl or algorithm.
12-06-2019
Yes, this does look like a duplicate of JDK-8217611. However, I wonder if the TLS code can somehow fall back to use the next preferred signature algorithm if the specified key cannot be used by the desired signature implementation. Or is it too late to fall back to a different signature algorithm at that point? I didn't close this as a duplicate of JDK-8217611 because if such scenario can be detected by TLS code and properly handled, then it can be addressed here,
22-05-2019
[~valeriep] Can you take a look at this one and see if it is an issue or a duplicate?
20-05-2019