United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-7157903 JSSE client sockets are very slow
JDK-7157903 : JSSE client sockets are very slow

Details
Type:
Bug
Submit Date:
2012-03-29
Status:
Closed
Updated Date:
2013-08-07
Project Name:
JDK
Resolved Date:
2012-05-05
Component:
security-libs
OS:
generic,solaris_10,windows_7
Sub-Component:
javax.net.ssl
CPU:
x86,sparc,generic
Priority:
P3
Resolution:
Fixed
Affected Versions:
6u30,7,7u4
Fixed Versions:
7u6 (b06)

Related Reports
Backport:
Backport:
Backport:
Backport:
Backport:
Backport:
Backport:
Duplicate:
Relates:
Relates:

Sub Tasks

Description
Testing JDK 7u4 java-based SSL clients to drive webserver traffic has regressed by two orders of magnitude. Our JSSE clients can drive >7400 requests/second with JDK 6 but only 124 requests/second with JDK 7u4 (and some earlier JDK 7 releases).

Although we are testing on JDK 7u4, I have tracked the regression to JDK 7u1 b06 when the code to split the first record data was added. If I run with -Djsse.enableCBCProtection=false then performance returns to normal.

The comment in that code indicates that we don't need to split if the TLS version is 1.1 or greater. In this case, the client is talking to a Java appserver also running with the JDK 7u4 JSSE libraries, so I'm not sure why it didn't negotiate an appropriate protocol to skip this. 

The other odd thing I notice is that we don't seem to be writing the appropriate amount of data. If I turn on SSL debugging with the default behavior, I see this:

Thread-0, WRITE: TLSv1 Application Data, length = 80  // FIRST REQUEST
Thread-0, READ: TLSv1 Application Data, length = 160
Thread-0, READ: TLSv1 Application Data, length = 32
Thread-0, READ: TLSv1 Application Data, length = 240 
Thread-0, WRITE: TLSv1 Application Data, length = 32  // SECOND REQUEST
Thread-0, WRITE: TLSv1 Application Data, length = 80 
Thread-0, READ: TLSv1 Application Data, length = 32
Thread-0, READ: TLSv1 Application Data, length = 160 
Thread-0, READ: TLSv1 Application Data, length = 32
Thread-0, READ: TLSv1 Application Data, length = 240 

The code is writing the same request repeatedly, so the data length of the first two requests should be the same; I'm not sure why in the second request we send a total of 112 bytes. But maybe that is just the way it works; I see that the server response in the first case (where it also didn't split the request) is a total 432 bytes but in the second case (it's second write, hence now splitting) also has an extra 32 bytes. When I add the -Djsse argument only to the client, the server still sends the extra 32 byte record but the client performance is back to normal.

                                    

Comments
WORK AROUND

If you are using LDAP/HTTPS, you can create a new default SSLSocketFactory to one that can preset the TcpNoDelay.  Then use the Security (not System) property to change the default property:

    Security.setProperty("ssl.SocketFactory.provider", "MySSLSocketFactory");

import java.io.IOException;
import java.net.*;
import javax.net.ssl.*;

public class MySSLSocketFactory extends SSLSocketFactory {

    private SSLSocketFactory sslsf;

    public MySSLSocketFactory() throws Exception {
        sslsf = (SSLSocketFactory) SSLContext.getDefault().getSocketFactory();
    }

    @Override
    public String[] getDefaultCipherSuites() {
        return sslsf.getDefaultCipherSuites();
    }

    @Override
    public String[] getSupportedCipherSuites() {
        return sslsf.getSupportedCipherSuites();
    }

    @Override
    public Socket createSocket() throws IOException {
        SSLSocket mySocket = (SSLSocket)sslsf.createSocket();
        mySocket.setTcpNoDelay(true);
        return mySocket;
    }

    @Override
    public Socket createSocket(Socket socket, String string,
            int i, boolean bln) throws IOException {
        SSLSocket mySocket =
                (SSLSocket) sslsf.createSocket(socket, string, i, bln);
        mySocket.setTcpNoDelay(true);
        return mySocket;
    }

    @Override
    public Socket createSocket(String string, int i)
            throws IOException, UnknownHostException {
        SSLSocket mySocket = (SSLSocket) sslsf.createSocket(string, i);
        mySocket.setTcpNoDelay(true);
        return mySocket;
    }

    @Override
    public Socket createSocket(String string, int i,
            InetAddress ia, int i1) throws IOException, UnknownHostException {
        SSLSocket mySocket = (SSLSocket) sslsf.createSocket(string, i, ia, i1);
        mySocket.setTcpNoDelay(true);
        return mySocket;
    }

    @Override
    public Socket createSocket(InetAddress ia, int i) throws IOException {
        SSLSocket mySocket = (SSLSocket) sslsf.createSocket(ia, i);
        mySocket.setTcpNoDelay(true);
        return mySocket;
    }

    @Override
    public Socket createSocket(InetAddress ia, int i,
            InetAddress ia1, int i1) throws IOException {
        SSLSocket mySocket = (SSLSocket) sslsf.createSocket(ia, i, ia1, i1);
        mySocket.setTcpNoDelay(true);
        return mySocket;
    }
}
                                     
2012-04-12
EVALUATION

The submitter of 7133330 has confirmed that this has fixed his issue.
                                     
2012-04-24
SQE has tests for this issue - JSSE/performance/bugs/bug7171614 
                                     
2013-05-02
EVALUATION

Nagle Algorithm (TCP_NODELAY) which affects small packets is false by default.  As a result, the kernel was throttling the JSSE output waiting for the peer's TCP ACK before sending the next packet.  Likely fix is to combine the small/output packets if TCP_NODELAY is not active.
                                     
2012-04-05
WORK AROUND

Use Socket.setTcpNoDelay(true).
                                     
2012-04-05
EVALUATION

Combining packets is the proposed option.  We will attach a ByteArrayOutputStream to the SSLSocket.  If the upper levels request packet splitting, we will hold that output in the BAOS until the second packet comes down, then recombine them and write to the socket.

One suggestion was to turn on TcpNoDelay for all SSLSockets by default, but that would be a sudden significant change in behavior in a minor release, and not an appropriate change.  However, user applications can certainly use that as a workaround until this change makes it into a release.

To followup on some Q's in the Description:

> The comment in that code indicates that we don't need to split if the TLS version 
> is 1.1 or greater. In this case, the client is talking to a Java appserver also 
> running with the JDK 7u4 JSSE libraries, so I'm not sure why it didn't negotiate 
> an appropriate protocol to skip this. 

SunJSSE only enables SSLv3/TLSv1 on the client side, but SSLv3->TLSv1.2 on the server.  If the client were set (sslSocket.setEnabledProtocols(String[]) to use v1.1/v1.2, it would have negotiated to one of these and skipped the whole split packets.

>I'm not sure why in the second request we send a total of 112 bytes.

Part of the overhead for block ciphersuites is some padding to round up to a complete block.

    header | appdata | MAC (messagedigest) | padding

The padding is likely making it appear that the same number of bytes are being written.
                                     
2012-04-10



Hardware and Software, Engineered to Work Together