JDK-8271199 : Mutual TLS handshake fails signing client certificate with custom sensitive PKCS11 key
  • Type: Bug
  • Component: security-libs
  • Sub-Component: java.security
  • Affected Version: 8,11,13,15,17,18
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2021-07-23
  • Updated: 2025-06-16
  • Resolved: 2021-10-25
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 17 JDK 18 Other
11.0.16Fixed 17.0.3Fixed 18 b21Fixed openjdk8u382,shenandoah8u382Fixed
Related Reports
Relates :  
JDK-8222937 - Cannot establish TLS connections in FIPS mode
Description
There is a use case with a custom PKCS11 provider (IAIK) which fails during the mutual TLS handshake.
TLS handshake fails to create a signature for the client certificate using sensitive PKCS11 RSA key.

EXPECTED BEHAVIOUR:
JDK selects custom RSASSA-PSS Signature implementation from the IAIK provider, signs client certificate, and completes TLS handshake

ACTUAL BEHAVIOUR:
JDK selects RSASSA-PSS Signature implementation from the SunRSASign provider and fails signing client certificate with the following stack trace:
  java.lang.UnsupportedOperationException: Prime P value is sensitive.
  at iaik.pkcs.pkcs11.provider.keys.IAIKPKCS11RsaPrivateKey.getPrimeP(Unknown Source)
  at java.base/sun.security.rsa.RSACore.crtCrypt(RSACore.java:176)
  at java.base/sun.security.rsa.RSACore.rsa(RSACore.java:130)
  at java.base/sun.security.rsa.RSAPSSSignature.engineSign(RSAPSSSignature.java:385)
  at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1404)
  at java.base/java.security.Signature.sign(Signature.java:712)
  at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:612)
  at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:764)
  at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:440)
  at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182)
  at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
  at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
  at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
  at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
  at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
  at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)

The implementation uses a private PKCS11 key from the IAIK provider :
http://javadoc.iaik.tugraz.at/pkcs11_provider/current/iaik/pkcs/pkcs11/provider/keys/IAIKPKCS11RsaPrivateKey.html

This issue happens because of SignatureScheme.getSigner() selects RSAPSSSignature signer from the SunRSASign provider and RSAPSSSignature.engineInitSign() successfully initialises signature object.

Stack trace to RSAPSSSignature.engineInitSign():
java.base/sun.security.rsa.RSAPSSSignature.engineInitSign(RSAPSSSignature.java:148)
  	at java.base/java.security.SignatureSpi.engineInitSign(SignatureSpi.java:167)
  	at java.base/java.security.Signature$Delegate.tryOperation(Signature.java:1318)
  	at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1270)
  	at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1382)
  	at java.base/java.security.Signature.initSign(Signature.java:683)
  	at java.base/java.security.Signature$1.initSign(Signature.java:147)
  	at java.base/sun.security.util.SignatureUtil.initSignWithParam(SignatureUtil.java:194)
  	at java.base/sun.security.ssl.SignatureScheme.getSigner(SignatureScheme.java:595)
  	at java.base/sun.security.ssl.SignatureScheme.getSignerOfPreferableAlgorithm(SignatureScheme.java:542)
  	at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:593)
  	at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:764)

As result, JDK does not try other signature providers. Later, RSAPSSSignature fails to sign the client certificate with the PKCS11 sensitive key.

This issue is possibly related to JDK-8226374 and JDK-8222937
Comments
Fix request [8u] I'd like to backport it to support custom PKCS11 providers. Clean backport from JDK11. sun/security/rsa jtreg test are passed
03-04-2023
A pull request was submitted for review. URL: https://git.openjdk.org/jdk8u-dev/pull/296 Date: 2023-04-03 07:08:54 +0000
03-04-2023
Fix request [11u] I'd like to backport it to support custom PKCS11 providers. Backport is not clean because of changes introduced by JDK-8023980 and JDK-8172366. These changes are not related to functionality of JDK-8271199 so merge is trivial. sun/security/rsa jtreg and custom test with IAIK provider are passed
21-04-2022
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk11u-dev/pull/1036 Date: 2022-04-19 12:33:22 +0000
19-04-2022
Fix request [17u] I'd like to backport it to support custom PKCS11 providers. The patch applies cleanly, all sun/security/rsa tests passed OK
12-02-2022
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk17u-dev/pull/157 Date: 2022-02-11 10:38:25 +0000
11-02-2022
Changeset: f6232982 Author: Alexey Bakhtin <abakhtin@openjdk.org> Committer: Yuri Nesterenko <yan@openjdk.org> Date: 2021-10-25 08:00:40 +0000 URL: https://git.openjdk.java.net/jdk/commit/f6232982b91cb2314e96ddbde3984836a810a556
25-10-2021