United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-7200295 CertificateRequest message is wrapping when using large numbers of Certs
JDK-7200295 : CertificateRequest message is wrapping when using large numbers of Certs

Details
Type:
Bug
Submit Date:
2012-09-21
Status:
Closed
Updated Date:
2014-02-05
Project Name:
JDK
Resolved Date:
2012-09-27
Component:
security-libs
OS:
generic
Sub-Component:
javax.net.ssl
CPU:
generic
Priority:
P3
Resolution:
Fixed
Affected Versions:
8
Fixed Versions:

Related Reports
Backport:
Backport:

Sub Tasks

Description
http://www.java.net/forum/topic/glassfish/glassfish/sslerrorrxmalformedcertrequest-two-way-ssl-authentication

In Glassfish 3.1.1 I have two-way ssl authentication and cacerts.jks has 498 certificates now.  When I have 516 entries (size 487KB) the server is starting but when I try to connect is rising:

    "Secure Connection Failed SSL received a malformed Certificate

I think the user is also seeing in OpenSSL:

    ssl_error_rx_malformed_cert_request

I tried to increase the allocated memory" the same result. The only way to make it to run is to delete one certificate. 

This is probably because there is an hardcoded limit in the protocol: The CertificateRequest message must specify the DNs of accepted CA. This DNs, all together, can occupy at the most 2^16-1= 65535 bytes, so if there are too many CAs, this limit can be encountered.

For reference, this is paragraph 7.4.4 of RFC 2246. It describes TLS 1.0, but there is little significant difference between TLS and SSL for what interest us.

                                    

Comments
EVALUATION

The root cause is a limitation of TLS unfortunately.  JSSE is not checking this when it comes to encoding,  is wrapping the short if the list gets above 64KB.   Thus is sending a truncated.  Need to throw an exception here if we wrap.

If the number of bytes read doesn't match up with the advertised header, the peer will fail on reads.
                                     
2012-09-21
changeset: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/a58585051c4b
                                     
2012-09-27
URL:   http://hg.openjdk.java.net/jdk8/tl/jdk/rev/a58585051c4b
User:  xuelei
Date:  2012-09-27 04:07:20 +0000

                                     
2012-09-27
Regression test is: sun/security/ssl/com/sun/net/ssl/internal/ssl/X509TrustManagerImpl/CertRequestOverflow.java
                                     
2012-09-27
URL:   http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/a58585051c4b
User:  lana
Date:  2012-10-12 18:08:25 +0000

                                     
2012-10-12



Hardware and Software, Engineered to Work Together