JDK-7200295 : CertificateRequest message is wrapping when using large numbers of Certs
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 8
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2012-09-21
  • Updated: 2016-04-22
  • Resolved: 2012-09-27
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7 JDK 8
7u40Fixed 8 b61Fixed
Related Reports
Relates :  
JDK-8153948 - sun/security/mscapi/ShortRSAKey1024.sh fails with "Field length overflow"
Relates :  
JDK-8154947 - Avoid server failure when list of authorities in CertificateRequest is too big
Description
http://www.java.net/forum/topic/glassfish/glassfish/sslerrorrxmalformedcertrequest-two-way-ssl-authentication

In Glassfish 3.1.1 I have two-way ssl authentication and cacerts.jks has 498 certificates now.  When I have 516 entries (size 487KB) the server is starting but when I try to connect is rising:

    "Secure Connection Failed SSL received a malformed Certificate

I think the user is also seeing in OpenSSL:

    ssl_error_rx_malformed_cert_request

I tried to increase the allocated memory" the same result. The only way to make it to run is to delete one certificate. 

This is probably because there is an hardcoded limit in the protocol: The CertificateRequest message must specify the DNs of accepted CA. This DNs, all together, can occupy at the most 2^16-1= 65535 bytes, so if there are too many CAs, this limit can be encountered.

For reference, this is paragraph 7.4.4 of RFC 2246. It describes TLS 1.0, but there is little significant difference between TLS and SSL for what interest us.
Comments
Regression test is: sun/security/ssl/com/sun/net/ssl/internal/ssl/X509TrustManagerImpl/CertRequestOverflow.java
27-09-2012
changeset: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/a58585051c4b
27-09-2012
EVALUATION The root cause is a limitation of TLS unfortunately. JSSE is not checking this when it comes to encoding, is wrapping the short if the list gets above 64KB. Thus is sending a truncated. Need to throw an exception here if we wrap. If the number of bytes read doesn't match up with the advertised header, the peer will fail on reads.
21-09-2012