JDK-8000186 : CertificateRequest message is wrapping when using large numbers of Certs
  • Type: Backport
  • Backport of: JDK-7200295
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 7u40,8
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • Submitted: 2012-09-21
  • Updated: 2013-09-12
  • Resolved: 2012-09-27
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7 JDK 8
7u40 b01Fixed 8Fixed
Description
http://www.java.net/forum/topic/glassfish/glassfish/sslerrorrxmalformedcertrequest-two-way-ssl-authentication

In Glassfish 3.1.1 I have two-way ssl authentication and cacerts.jks has 498 certificates now.  When I have 516 entries (size 487KB) the server is starting but when I try to connect is rising:

    "Secure Connection Failed SSL received a malformed Certificate

I think the user is also seeing in OpenSSL:

    ssl_error_rx_malformed_cert_request

I tried to increase the allocated memory" the same result. The only way to make it to run is to delete one certificate. 

This is probably because there is an hardcoded limit in the protocol: The CertificateRequest message must specify the DNs of accepted CA. This DNs, all together, can occupy at the most 2^16-1= 65535 bytes, so if there are too many CAs, this limit can be encountered.

For reference, this is paragraph 7.4.4 of RFC 2246. It describes TLS 1.0, but there is little significant difference between TLS and SSL for what interest us.

Comments
changeset: http://hg.openjdk.java.net/jdk7u/jdk7u-dev/jdk/rev/b76b6f452d29
27-09-2012