Bug ID: JDK-8272351 Mutual TLS has stopped working - client certificates using RSA SHA256 not found

JDK-8272351 : Mutual TLS has stopped working - client certificates using RSA SHA256 not found

Type: Bug
Component: security-libs
Sub-Component: javax.net.ssl
Affected Version: 8u281,11.0.10,18

Priority: P3
Status: In Progress
Resolution: Unresolved

Submitted: 2021-08-05
Updated: 2022-01-20

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

Other
tbdUnresolved

Related Reports

Relates :	JDK-8206925 - Support the certificate_authorities extension
Relates :	JDK-8226374 - Restrict TLS signature schemes and named groups

Description

ADDITIONAL SYSTEM INFORMATION :
Dockerised version of Amazon Corretto.

Corretto 11.0.11 and prior versions all work fine
Corretto 11.0.12 fails to find client certificate

A DESCRIPTION OF THE PROBLEM :
Mutual TLS using client and server certs created using rsa_pss_rsae_sha256.
Java 11 up to 11.0.11 worked fine for both server and client (both X509Authentication.createServerPossession() and X509Authentication.createClientPossession() returned successfully).
Updating the runtime to Java 11.0.12, the server cert is returned successfully but the client gets null back from X509Authentication.createClientPossession(). As such, the Mutual TLS handshake does not complete as the server doesn't get the required client cert and returns "bad certificate".

The change that seems to have caused the issue is this one: https://github.com/openjdk/jdk11u/commit/b8741052c5c86eaf4b819cfd84543cf03285033d

REGRESSION : Last worked in version 11

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Create server and client certs (2048-bit RSA with SHA-256)

Create TLS 1.3 Web service that requires client authentication (this doesn't have to be Java but helps logging - I just use a dummy Spring Boot app to return static information)

Create Java app which calls the above Web service, ensuring CA certs signing the server's cert are in the trust store and the named client private key cert is in the key store.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Successful Mutual TLS handshake and response.
ACTUAL -
Bad certificate error and TLS tunnel not established.


For completeness, here's the client logs when trying to respond to a CertificateRequest from the server:
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.095 BST|X509Authentication.java:246|No X.509 cert selected for EC`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.096 BST|CertificateMessage.java:1084|Unavailable authentication scheme: ecdsa_secp256r1_sha256`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.099 BST|X509Authentication.java:246|No X.509 cert selected for EC`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.099 BST|CertificateMessage.java:1084|Unavailable authentication scheme: ecdsa_secp384r1_sha384`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.100 BST|X509Authentication.java:246|No X.509 cert selected for EC`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.100 BST|CertificateMessage.java:1084|Unavailable authentication scheme: ecdsa_secp521r1_sha512`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.101 BST|X509Authentication.java:246|No X.509 cert selected for RSA`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.102 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pss_rsae_sha256`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.102 BST|X509Authentication.java:246|No X.509 cert selected for RSA`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.102 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pss_rsae_sha384`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.103 BST|X509Authentication.java:246|No X.509 cert selected for RSA`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.103 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pss_rsae_sha512`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.103 BST|X509Authentication.java:246|No X.509 cert selected for RSASSA-PSS`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.104 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pss_pss_sha256`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.104 BST|X509Authentication.java:246|No X.509 cert selected for RSASSA-PSS`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.104 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pss_pss_sha384`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.105 BST|X509Authentication.java:246|No X.509 cert selected for RSASSA-PSS`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.105 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pss_pss_sha512`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.106 BST|X509Authentication.java:246|No X.509 cert selected for RSA`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.107 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pkcs1_sha256`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.110 BST|X509Authentication.java:246|No X.509 cert selected for RSA`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.110 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pkcs1_sha384`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.111 BST|X509Authentication.java:246|No X.509 cert selected for RSA`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.111 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pkcs1_sha512`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.112 BST|X509Authentication.java:246|No X.509 cert selected for EC`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.112 BST|CertificateMessage.java:1084|Unavailable authentication scheme: ecdsa_sha1`
`javax.net.ssl|ALL|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.112 BST|X509Authentication.java:246|No X.509 cert selected for RSA`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.116 BST|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pkcs1_sha1`
`javax.net.ssl|WARNING|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.116 BST|CertificateMessage.java:1094|No available authentication scheme`
`javax.net.ssl|DEBUG|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.117 BST|CertificateMessage.java:1106|No available client authentication scheme`
`javax.net.ssl|DEBUG|1D|http-nio-8008-exec-1|2021-08-05 10:19:24.117 BST|CertificateMessage.java:1140|Produced client Certificate message (`
`"Certificate": {`
` "certificate_request_context": "",`
` "certificate_list": [`
` ]`
`}`
`)`

(i.e. no client certificate found when, in Java 11.0.11, it was found OK)

Whereas, here's the server getting the server cert on the same Java 11.0.12.
`javax.net.ssl|ALL|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:47.995 BST|X509Authentication.java:301|No X.509 cert selected for EC`
`javax.net.ssl|WARNING|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:47.995 BST|CertificateMessage.java:1084|Unavailable authentication scheme: ecdsa_secp256r1_sha256`
`javax.net.ssl|ALL|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:47.996 BST|X509Authentication.java:301|No X.509 cert selected for EC`
`javax.net.ssl|WARNING|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:47.997 BST|CertificateMessage.java:1084|Unavailable authentication scheme: ecdsa_secp384r1_sha384`
`javax.net.ssl|ALL|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:47.998 BST|X509Authentication.java:301|No X.509 cert selected for EC`
`javax.net.ssl|WARNING|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:47.998 BST|CertificateMessage.java:1084|Unavailable authentication scheme: ecdsa_secp521r1_sha512`
`javax.net.ssl|DEBUG|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:47.999 BST|SunX509KeyManagerImpl.java:392|matching alias: <redacted>`
`javax.net.ssl|DEBUG|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:48.001 BST|StatusResponseManager.java:763|Staping disabled or is a resumed session`
`javax.net.ssl|ALL|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:48.002 BST|CertStatusExtension.java:1112|Stapling is disabled for this connection`
`javax.net.ssl|DEBUG|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:48.002 BST|SSLExtensions.java:260|Ignore, context unavailable extension: status_request`
`javax.net.ssl|DEBUG|13|https-jsse-nio-443-exec-1|2021-08-05 09:34:48.007 BST|CertificateMessage.java:1022|Produced server Certificate message (`
`"Certificate": {`
`  "certificate_request_context": "",`
`  "certificate_list": [  `
`  {`
`    "certificate" : {`
`      "version"            : "v3",`
`      "serial number"      : "<redacted>",`
`      "signature algorithm": "SHA256withRSA",`
`      "issuer"             : "<redacted>",`
`      "not before"         : "2020-10-28 11:20:20.000 GMT",`
`      "not  after"         : "2030-10-26 12:20:20.000 BST",`
`      "subject"            : "<redacted>",`
`      "subject public key" : "RSA",`
`      "extensions"         : [`
`        {`
`          ObjectId: 2.16.840.1.113730.1.13 Criticality=false`
`        },`
`        {`
`          ObjectId: 2.5.29.19 Criticality=false`
`          BasicConstraints:[`
`            CA:false`
`            PathLen: undefined`
`          ]`
`        },`
`        {`
`          ObjectId: 2.5.29.37 Criticality=false`
`          ExtendedKeyUsages [`
`            serverAuth`
`          ]`
`        },`
`        {`
`          ObjectId: 2.5.29.15 Criticality=true`
`          KeyUsage [`
`            DigitalSignature`
`            Key_Encipherment`
`          ]`
`        },`
`        {`
`          ObjectId: 2.16.840.1.113730.1.1 Criticality=false`
`          NetscapeCertType [`
`             SSL server`
`          ]`
`        },`
`        {`
`          ObjectId: 2.5.29.17 Criticality=false`
`          SubjectAlternativeName [`
`            DNSName: <redacted>`
`          ]`
`        }`
`      ]}`
`    "extensions": {`
`      <no extension>`
`    }`
`  },`
`]`
`}`
`)`

The rsa_pss_rsae_sha256 algorithm is there as that's what is used by the test server using the exact same runtime. The keystore is good and I can see that the named client private key certificate is in there and loads OK.

All that's different is that, moving to Java 11.0.12 from 11.0.11 causes the issue.

(I did see from the above GitHub commit that there's a new system property jdk.tls.client.enableCAExtension which I tried setting. Made no difference)

CUSTOMER SUBMITTED WORKAROUND :
Use Java 11.0.11 for now.

FREQUENCY : always

Comments

Response from the submitter: OK that does indeed seem to resolve the issue. I’ve also tried removing the specific client certificate from the server truststore and only having the two CA certs there, and that works too. So what this says is that, going forward, a server requiring Mutual TLS (i.e. client authentication too) cannot trust individual client certificates but must trust all client certificates from a client CA. This is a change from what was previously allowed and will be a change to how we set up trust on our server. That’s fine as a workaround but please can you tell me where this will be published as a change as I can’t see that anywhere in the JDK release notes as a direction to developers?
02-09-2021
Requested the submitter try the above suggestion.
02-09-2021
Can you please ask the customer to add mutualtlsclientCA certificate to server server-truststore.p12.
02-09-2021
"certificate_authorities" extension introduced in JDK15 through JDK-8206925. We backported JDK-8206925 to 11.0.10-oracle JDK. In 11.0.9-oracle JDK, the client produces the certificate with the parameters mentioned in the CertificateRequest message from the server. After JDK-8206925 fix, certificate_authorities info was added to CertificateRequest parameters list and the client looks for the certificates issued by the authorities mentioned in the certificate_authorities. In this case, server-truststore.p12 has the following 2 certs, "Owner: CN=client, OU=MutualTLSIssueReproduction, O=XXXX, L=XXXX, ST=XXXX, C=UK Issuer: CN=mutualtlsclientCA, OU=MutualTLSIssueReproduction, O=XXXX, L=XXXX, ST=XXXX, C=UK" "Owner: CN=mutualtlsserverCA, OU=MutualTLSIssueReproduction, O=XXXX, L=XXXX, ST=XXXX, C=UK Issuer: CN=mutualtlsserverCA, OU=MutualTLSIssueReproduction, O=XXXX, L=XXXX, ST=XXXX, C=UK" and the server is sending subject/owner details of each certificate in certificate_authorities extension, 2 entries in this case. CN=client, OU=MutualTLSIssueReproduction, O=XXXX, L=XXXX, ST=XXXX, C=UK CN=mutualtlsserverCA, OU=MutualTLSIssueReproduction, O=XXXX, L=XXXX, ST=XXXX, C=UK] Client finds the list of certificates issued by the entries in the certificate_authorities extension and share it to the server. In this case, client is not able to find certs issued by client/ mutualtlsserverCA in its keystore.
02-09-2021
The observations on Windows 10: JDK 11.0.9.0.4: Passed. JDK 11.0.10-ea+1: Failed, got bad cert error. JDK 18ea=1: Failed.
12-08-2021
Additional information from the submitter: Table of versions of JDK running on Windows 10 64-bit as follows: • 11.0.11 – gives the bad cert error • 11.0.10 – gives the bad cert error • 11.0.9 – works just fine • 11.0.8 – works just fine • 11.0.7 – works just fine In terms of looking at what code changes merged into an OpenJDK could have caused this, I found this potential candidate (please also see comments I added there, especially a possible bug in one class) 8206925: Support the certificate_authorities extension · openjdk/jdk11u@b874105 (github.com)
12-08-2021
Additional information from the submitter: This change also might be worth looking at as it restricts signature schemes for TLS for particular uses… and was introduced at the same 11.0.12 of OpenJDK. Again, was this merged into Oracle JDK 11.0.10? 8226374: Restrict TLS signature schemes and named groups · openjdk/jdk11u@879d119 (github.com)
12-08-2021
Requested a simple reproducer from the submitter.
07-08-2021