JDK-8214129 : SSL session resumption/SNI with TLS1.2 causes StackOverflowError
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 11,12
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2018-11-20
  • Updated: 2023-07-26
  • Resolved: 2018-12-07
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 12 JDK 8 Other
11.0.3-oracleFixed 12 b24Fixed 8u261Fixed openjdk8u272Fixed
Related Reports
Duplicate :  
Relates :  
Relates :  
Description
ADDITIONAL SYSTEM INFORMATION :
Tested on Fedora 29 and Debian Jessie
With Java 11.0.1 (OpenJDK + Oracle)

A DESCRIPTION OF THE PROBLEM :
We came across a major Java 11 bug within the SSL session resumption/Server name indication code
causing StackOverflowErrors when using the built-in HTTPClient (or HttpURLConnection) 
together with HTTPS and TLS version 1.2. 

We could observe the problem on production systems which are making lots of
HTTPS service calls to clustered endpoints. Depending on the amount of requests
and the thread stack size, the StackOverflowErrors are showing up after a few days
uptime and we have to restart the JVMs.

I could track down the problem to the class sun.security.ssl.SSLSessionImpl,
where a list of requestedServerNames from the HandshakeContext is put into an unmodifiable list
again and again, when the same session is resumed, thus ending up with a nested
list exceeding the thread stack size, when accessed in 
sun.security.ssl.ServerNameExtension$CHServerNameProducer.produce().

I attached some test code, which can reproduce the problem using raw sockets.
The bug seems to be new in Java 11 and only applying to TLS 1.2.

REGRESSION : Last worked in version 10

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Make thousands of HTTPS requests using HTTPClient/HttpURLConnections to a clustered server using TLSv1.2 (probably it's important that no session tickets are used and the endpoint is clustered, so that you get session id cache misses on the server and the client creates new ones).


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Should just work.
ACTUAL -
Exception in thread "Thread-1" java.lang.StackOverflowError
	at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1042)
	at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1041)
...
	at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1042)
	at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1041)
	at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1042)
	at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1041)
	at java.base/sun.security.ssl.ServerNameExtension$CHServerNameProducer.produce(ServerNameExtension.java:228)
	at java.base/sun.security.ssl.SSLExtension.produce(SSLExtension.java:532)
	at java.base/sun.security.ssl.SSLExtensions.produce(SSLExtensions.java:228)
	at java.base/sun.security.ssl.ClientHello$ClientHelloKickstartProducer.produce(ClientHello.java:648)
	at java.base/sun.security.ssl.SSLHandshake.kickstart(SSLHandshake.java:515)
	at java.base/sun.security.ssl.ClientHandshakeContext.kickstart(ClientHandshakeContext.java:104)
	at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:228)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
	at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:716)
	at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:970)
	at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:942)

---------- BEGIN SOURCE ----------
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.SocketTimeoutException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

import javax.net.ssl.SNIHostName;
import javax.net.ssl.SNIMatcher;
import javax.net.ssl.SNIServerName;
import javax.net.ssl.SSLParameters;
import javax.net.ssl.SSLServerSocket;
import javax.net.ssl.SSLServerSocketFactory;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;

/**
 * Test showing StackOverflowError within
 * sun.security.ssl.ServerNameExtension$CHServerNameProducer.produce().
 * 
 * The test should be started with minimal thread stack size and increased
 * stacktrace depth:
 * 
 * <pre>
 * java -Xss140k -XX:MaxJavaStackTraceDepth=2000 SNIBugTest
 * </pre>
 * 
 * Using these settings the test should fail after about 778 iterations.
 * 
 * By default the test uses the hostname localhost and localhost.localdomain as
 * SNI name. By default it creates a keystore with a self-signed certificate.
 * You can change the host names and provide custom javax.net.ssl.XXX settings
 * if you want to test with an alternative domain or another
 * keystore/truststore.
 */
public class SNIBugTest {

    static String hostName = "localhost";
    static String sniHostName = "localhost.localdomain";

    static int maxRequests = 1000;

    static volatile int serverPort;
    static AtomicInteger requestCount = new AtomicInteger();

    public static void main(String[] args) throws Exception {

        if (System.getProperty("javax.net.ssl.keystore") == null) {
            createKeystore();
            System.setProperty("javax.net.ssl.keyStore", "testkeystore");
            System.setProperty("javax.net.ssl.keyStorePassword", "passphrase");
            System.setProperty("javax.net.ssl.trustStore", "testkeystore");
            System.setProperty("javax.net.ssl.trustStorePassword",
                    "passphrase");
        }

        ServerThread server = new ServerThread();
        server.start();

        while (serverPort == 0) {
            Thread.sleep(100);
        }

        ClientThread client = new ClientThread();
        client.start();
        client.join();

        server.interrupt();
    }

    static void createKeystore() throws Exception {

        ProcessBuilder builder = new ProcessBuilder("keytool", "-genkey",
                "-alias", "dummy", "-keyalg", "RSA", "-keysize", "2048",
                "-sigalg", "SHA256withRSA", "-validity", "365", "-keypass",
                "passphrase", "-keystore", "testkeystore", "-storepass",
                "passphrase", "-dname", "CN=localhost.localdomain, OU=Dummy,"
                + " O=Dummy, L=Cupertino, ST=CA, C=US");
        builder.redirectErrorStream(true);
        Process process = builder.start();
        int exitCode = process.waitFor();
        if (exitCode != 0) {
            BufferedReader reader = new BufferedReader(
                    new InputStreamReader(process.getInputStream()));
            String line;
            while ((line = reader.readLine()) != null) {
                System.out.println(line);
            }
        }
    }

    static class ClientThread extends Thread {

        @Override
        public void run() {
            SSLSocketFactory factory = (SSLSocketFactory)
                    SSLSocketFactory.getDefault();
            for (int i = 0; i < maxRequests; i++) {
                try (SSLSocket sslSocket = (SSLSocket) factory
                        .createSocket(hostName, serverPort)) {
                    SNIHostName serverName = new SNIHostName(sniHostName);
                    List<SNIServerName> serverNames = new ArrayList<>(1);
                    serverNames.add(serverName);
                    SSLParameters params = sslSocket.getSSLParameters();
                    params.setServerNames(serverNames);
                    sslSocket.setSSLParameters(params);
                    OutputStream out = sslSocket.getOutputStream();
                    InputStream in = sslSocket.getInputStream();
                    out.write(0);
                    in.read();
                } catch (IOException x) {
                    x.printStackTrace();
                }
            }
        }
    }

    static class ServerThread extends Thread {

        @Override
        public void run() {

            SSLServerSocketFactory factory = (SSLServerSocketFactory)
                    SSLServerSocketFactory.getDefault();
            try (SSLServerSocket serverSocket = (SSLServerSocket) factory
                    .createServerSocket(0)) {
                serverPort = serverSocket.getLocalPort();
                serverSocket.setSoTimeout(1000);
                // force TLS version 1.2
                serverSocket.setEnabledProtocols(new String[] { "TLSv1.2" });
                while (!isInterrupted()) {
                    try (SSLSocket socket = 
                            (SSLSocket) serverSocket.accept()) {
                        SNIMatcher matcher = SNIHostName
                                .createSNIMatcher(sniHostName);
                        Collection<SNIMatcher> matchers = new ArrayList<>(1);
                        matchers.add(matcher);
                        SSLParameters params = serverSocket.getSSLParameters();
                        params.setSNIMatchers(matchers);
                        System.out.println(requestCount.incrementAndGet());
                        OutputStream out = socket.getOutputStream();
                        InputStream in = socket.getInputStream();
                        int data = in.read();
                        out.write(data);
                        // simulate failed session lookup on clustered system
                        socket.getSession().invalidate();
                    } catch (SocketTimeoutException x) {
                        continue;
                    } catch (IOException x) {
                        x.printStackTrace();
                    }
                }
            } catch (IOException x) {
                x.printStackTrace();
            }
        }
    }

}
---------- END SOURCE ----------

FREQUENCY : always



Comments
This does not affect OpenJDK 8u, because both affected files were brought from 11.0.7 by JDK-8245468 when TLS 1.3 was backported.
14-10-2020

Fix Request This is an important bugfix that should go back to jdk11u. The patch applies cleanly.
15-02-2019

URL: http://hg.openjdk.java.net/jdk/jdk/rev/b0e751c70385 User: jnimeh Date: 2018-12-07 06:07:02 +0000
07-12-2018

There are several places that creating new unmodifiable collections from unmodified collections. We might want to use unmodifiable collections more carefully. If a collection is unmodifiable, it does not make sense to create a new unmodifiable collection from it. We may want to avoid: Collection object = Collections.unmodifiableCollection( unmodifiableCollection);
28-11-2018

I could reproduce the issue on ubuntu 14.0.4, but not on windows 7. JDK 11 GA - Fail JDK 11.0.1 - Fail JDK 12-ea+13 - Fail The output is attached.
20-11-2018