JDK-8072046 : REGRESSION: Difference in behavior between 8u25 and 7u72 when SSL certificates have the RSASSA-PSS signature algorithm
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 8u20,8u25,8u31
  • Priority: P2
  • Status: Resolved
  • Resolution: Not an Issue
  • Submitted: 2015-01-31
  • Updated: 2016-09-14
  • Resolved: 2015-03-11
Related Reports
Relates :  
JDK-8146293 - Add support for RSASSA-PSS Signature algorithm
Description
Have noticed an unexpected difference in behavior between 8u25 and 8u31 for 
SSL.  Client and server certificate s attached.

The attached certificate works fine for 8u25, but in 8u31 we get the 
following in the ssl log output:

%% Invalidated:  [Session-1, TLS_RSA_WITH_AES_128_CBC_SHA]
main, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
main, WRITE: TLSv1.2 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 03 00 02 02 2E                               .......
main, called closeSocket()
main, handling exception: javax.net.ssl.SSLHandshakeException: 
java.security.cert.CertificateException: Certificates does not conform to 
algorithm constraints
main, called close()
main, called closeInternal(true)
main, called close()
main, called closeInternal(true)
main, called close()
main, called closeInternal(true)
Comments
I am closing as "Not an Issue" unless escalated.
11-03-2015
Customer likely has third party providers (PSS and to a lesser degree ECC(entrust)) on client/server. I'm not able to test further without help. There is likely a custom X509TrustManager in play (not an ExtendedX509TrustManger). However, if there was an ExtendedX509TrustManager, this would probably also fail for the same reason but in a different part of the code. JDK 8 uses TLSv1.2 by default which drags in the AlgorithmConstraints for the Signature Hello Extension Constaint checks. Since JDK 7 defaults to TLSv1, and doesn't have AlgorithmConstraints, it won't have that check. The third party providers will likely allow the certs to work as desired in JDK 7. Back to JDK 8, the core problem is that within the Signature Hello Extension AlgorithmConstraint check, we don't recognize "1.2.840.113549.1.1.10" for PSS, so the "<digest>with<encryption>" field is not being returned and parsing out correctly, which is causing the Constraints check failure. Given the comment about the IETF efforts, we might want to add the AlgorithmId.java to recognize PSS algorithm, and maybe also longer term support PSS mode, but I don't think there is anything we can do for the customer at the JSSE level other than to say PSS certs are not supported at the IETF TLS level. Customer should not use PSS certs.
04-02-2015
A couple other items of note: RFC 5246: 7.4.2. Server Certificate Note that there are certificates that use algorithms and/or algorithm combinations that cannot be currently used with TLS. For example, a certificate with RSASSA-PSS signature key (id-RSASSA-PSS OID in SubjectPublicKeyInfo) cannot be used because TLS defines no corresponding signature algorithm. This may be changed in TLSv1.3. There is some discussion in the IETF TLS working group about this. http://ietf.10.n7.nabble.com/Using-RSA-PSS-in-TLS-td387873.html
04-02-2015
It looks like the customer is using a 3rd party providers that provides a PSS implementation and also a ECC implementation. I notice in the "command window log.txt" (client) the ECC curve names are showing up as: Extension elliptic_curves, curve names: {com.entrust.toolkit.security.crypto.ec.EcParameterSpecWithName@2842bbc4, 1.3.132.0.1, 1.3.132.0.15, com.entrust.toolkit.security.crypto.ec.EcParameterSpecWithName@7c6b7a06, com.entrust.toolkit.securi instead of the usual secp256r1 as in stdout.log, and the certificates are showing up as: Signature algorithm: id-RSASSA-PSS with parameter which we don't have. I also believe the customer has a custom TrustManager which only extends X509TrustManager. Can you confirm all this? That would explain why the provided keystore does not work in a stock JDK environment. Are you able to duplicate this in house? Can you please provide an environment I can use for successful/failure testing using 7uXX/8 respectively? I have simple test apps that take a keystore/password and will do a simple connection, but I don't have the customers providers to check with. On the JSSE side, we may need to update the list of acceptable signature fields. Note this AlgorithmConstraint check is only done in JDK 8 and on TLS v1.2 and above. (TLSv1.2 is also not on by default on JDK 7.) One workaround is to use TLSv1.1 instead.
04-02-2015
Here are the details, the AlgorithmConstraints are getting checked for the allowed Signature Algorithm as part of the Hello Extensions in RFC 5246:7.4.1.4.1. Because 1.2.840.113549.1.1.10 doesn't map to an algorithm name of <digest>with<encryption>and<mgf> in AlgorithmId.java:890, it is failing the Constraints check and thus failing the overall connection. RSASSA-PSS was introduced in PKCS#1 v2.1, and doesn't exist in previous versions (v1.5/2), and apparently we SSLContextImpl:131. A customized TrustManager which only implements X509TrustManager must be used, in order for the AbstractTrustManagerWrapper to wrap it. Our default X509TM implement X509ExtendedTrustManager, so they won't see this problem. SSLContextImpl:937 When checkServerTrusted is called, it calls to checkAdditionalTrust() SSLContextImpl:1006 Algorithm constraints are only checked in TLSv1.2. Algorithm constraints are new to JDK 8. SSLContextImpl:1051 The underlying exception that is getting swallowed is: Algorithm constraints check failed: 1.2.840.113549.1.1.10 which throws an exception at 1055, but doesn't include the cause/reason. IMO, we should initCause() here. Going further down: AlgorithmChecker:185 gets the Signature name, but comes back with OID since there is no name to map to. AlgorithmChecker:197 the constraints.permits fails because SSLAlgorithmConstraints:238 looks for "and" but can't find it, thus the textual parsing to get a <digest>with<encryption> fails, and the loop for checking the supported algorithm fails. It's not clear yet why this would pass in 7u72, I'm looking into that now.
04-02-2015
log files and certificate plus keys are attached
04-02-2015
Notes: TLSv1.2 used in failure (JDK8), TLSv1 used in success (JDK7). These are the default protocols. Client is likely using a custom trust manager.
04-02-2015
I see the client log has been added, I'll take a look. Would it be possible to get a keystore with similar RSASSA-PSS keys in case we need to step through it?
03-02-2015
marking incomplete until logs are received (javax.net.debug=all) - Ideally logs should be obtained from server and client end.
03-02-2015
Earlier indicated the problem appeared in 8u31. That is not correct. The problem has been reproduced in 8u25 and 8u20. 7u72 does not have the issue.
03-02-2015
We need more information to debug the problem. See previous comment for requested information.
03-02-2015
More information from the CAP member - The issue occurs only when the SSL certificates have the RSASSA-PSS signature algorithm. SSL certificates that use regular RSA algorithms work OK.
02-02-2015