A DESCRIPTION OF THE PROBLEM :
Client SSL Session implementation causes StackOverflowError when the remote sever invalidates session cache many times.
When a new instance of SSLSessionImpl is constructed, SSLSessionImpl.requestedServerNames field is initialized with list SNIServerName provided by HandshakeContext.getRequestedServerNames() method, wrapped by Collections.unmodifiableList().
The same list instance is retrieved in ServerNameExtension$CHServerNameProducer.produce() method by accessing SSLSessionImpl.requestedServerNames field and stored in ClientHandshakeContext.requestedServerNames field.
It results in creating chain of list wrappers around original list. When this chain is long enough (usually thousands of wrappers are needed) StackOverflowError is thrown when accessing a value from the list e.g. via get() method.
java.lang.StackOverflowError: null
at java.base/java.util.Collections$UnmodifiableList.get(Collections.java:1308)
at java.base/java.util.Collections$UnmodifiableList.get(Collections.java:1308)
....
at java.base/java.util.Collections$UnmodifiableList.get(Collections.java:1308)
at java.base/sun.security.ssl.ServerNameExtension$SHServerNameConsumer.consume(ServerNameExtension.java:496)
at java.base/sun.security.ssl.SSLExtension.consumeOnLoad(SSLExtension.java:542)
at java.base/sun.security.ssl.SSLExtensions.consumeOnLoad(SSLExtensions.java:164)
at java.base/sun.security.ssl.ServerHello$T12ServerHelloConsumer.consume(ServerHello.java:1084)
at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(ServerHello.java:990)
at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.consume(ServerHello.java:872)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
at com.sun.mail.util.SocketFetcher.configureSSLSocket(SocketFetcher.java:626)
at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:400)
at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:238)
at com.sun.mail.iap.Protocol.<init>(Protocol.java:134)
at com.sun.mail.imap.protocol.IMAPProtocol.<init>(IMAPProtocol.java:131)
at com.sun.mail.imap.IMAPStore.newIMAPProtocol(IMAPStore.java:763)
at com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:698)
at javax.mail.Service.connect(Service.java:366)
...
Adding an additional link to the list wrappers chain happens when the SSL session is resumed in ServerHello$T12ServerHelloConsumer.consume() method, but only when serverHello.sessionId is not equal to chc.resumingSession.getSessionId() (where chc is ClientHandshakeContext).
In such case the chc.resumingSession is invalidated and nullified, false assigned to chc.isResumption flag and later new SSLSessionImpl is created and chc is passed as first constructor argument:
chc.handshakeSession = new SSLSessionImpl(chc, ...);
As explained at the beginning, in SSLSessionImpl constructor, the list of requested server names (retrieved from chc.getRequestedServerNames()) is wrapped again by Collections.unmodifiableList() and this is how chain builds up.
To fix the issue it is enough to brake a chain e.g. by creating a copy of the list obtained from HandshakeContext.getRequestedServerNames() in SSLSessionImpl constructor before such list is wrapped by Collections.unmodifiableList() and assigned to requestedServerNames field.
REGRESSION : Last worked in version 8u191
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Reproducing this issue is not trivial. It requires connecting to server which occasionally invalidates the SSL session. The bug was spotted while reconnecting to imap.google.com using 993 port. This sever invalidates SSL session occasionally but predictable time.
It also takes a lot of time to build so long chain of list wrappers that it will cause StackOverflowError. To reproduce it in test environment, byteman was used to introspect SSL implementation internals and detect when a new link to the chain is added.
FREQUENCY : always