JDK-8214098 : sun.security.ssl.HandshakeHash.T12HandshakeHash constructor check backwards.
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 11,12
  • Priority: P2
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2018-11-14
  • Updated: 2020-11-23
  • Resolved: 2018-12-10
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 12 JDK 8 Other
11.0.6-oracleFixed 12 b24Fixed 8u261Fixed openjdk8u272Fixed
Description
ADDITIONAL SYSTEM INFORMATION :
openjdk version "11" 2018-09-25
OpenJDK Runtime Environment (build 11+24-Ubuntu-116.04.1)
OpenJDK 64-Bit Server VM (build 11+24-Ubuntu-116.04.1, mixed mode, sharing)

A DESCRIPTION OF THE PROBLEM :
In the constructor for T12HandshakeHash in HandshakeHash:

            MessageDigest md =
                    JsseJce.getMessageDigest(cipherSuite.hashAlg.name);
            if (md instanceof Cloneable) {
                transcriptHash = new CloneableHash(md);
                this.baos = null;
            } else {
                transcriptHash = new NonCloneableHash(md);
                this.baos = new ByteArrayOutputStream();
            }

This is clearly backwards. It guarantees that CloneableHash.archived() is called when T12HandshakeHash.archived() is called which guarantees an UnsupportedOperationException will be thrown. It looks like the same error has been made in T13Handshake as well.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Issue with the code can be seen by inspection.


CUSTOMER SUBMITTED WORKAROUND :
No work around seems possible.
Comments
Fix Request (11u) This fixes the TLS handshake bug and keeps codebases in sync (I see 11.0.6-oracle). Patch applies cleanly to 11u, passes tier1 and tier2 tests.
04-09-2019
From submitter: Here is the stack trace that was reported to us, the exception thrown was UnsupportedOperationException archived:616, HandshakeHash$CloneableHash (sun.security.ssl) archived:546, HandshakeHash$T12HandshakeHash (sun.security.ssl) archived:188, HandshakeHash (sun.security.ssl) <init>:581, CertificateVerify$T12CertificateVerifyMessage (sun.security.ssl) produce:740, CertificateVerify$T12CertificateVerifyProducer (sun.security.ssl) produce:436, SSLHandshake (sun.security.ssl) consume:173, ServerHelloDone$ServerHelloDoneConsumer (sun.security.ssl) consume:392, SSLHandshake (sun.security.ssl) dispatch:444, HandshakeContext (sun.security.ssl) run:1065, SSLEngineImpl$DelegatedTask$DelegatedAction (sun.security.ssl) run:1052, SSLEngineImpl$DelegatedTask$DelegatedAction (sun.security.ssl) doPrivileged:-1, AccessController (java.security) run:999, SSLEngineImpl$DelegatedTask (sun.security.ssl) runDelegatedTasks:1435, SslHandler (io.netty.handler.ssl) unwrap:1343, SslHandler (io.netty.handler.ssl) decodeJdkCompatible:1177, SslHandler (io.netty.handler.ssl) decode:1221, SslHandler (io.netty.handler.ssl) decodeRemovalReentryProtection:489, ByteToMessageDecoder (io.netty.handler.codec) callDecode:428, ByteToMessageDecoder (io.netty.handler.codec) channelRead:265, ByteToMessageDecoder (io.netty.handler.codec) The failure was reported against OpenJDK, the same failure was reported against the Oracle JVM as well. If it's any help when I was checking the source for OpenJDK it was clear by inspection that the code in HandshakeHash was incorrect. Here's a more complete description of the issue and the fix which I hope will answer any other questions you might have. Line numbers are based on: https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.base/share/classes/sun/security/ssl/HandshakeHash.java The issue reported was in the constructor of T12HandshakeHash on lines 516 to 526 where the code is: T12HandshakeHash(CipherSuite cipherSuite) { MessageDigest md = JsseJce.getMessageDigest(cipherSuite.hashAlg.name); if (md instanceof Cloneable) { transcriptHash = new CloneableHash(md); this.baos = null; } else { transcriptHash = new NonCloneableHash(md); this.baos = new ByteArrayOutputStream(); } } Unfortunately this guarantees that any attempt to use a Clonable message digest, such as those provided by the BC/BCFIPS providers, and most others, will result in the throwing of new UnsupportedOperationException("Not supported yet."), when the archived() method on the class is called. The exception is thrown by CloneableHash on line 616 as it does not maintain a copy of what is fed into it. NonCloneableHash on the other hand does maintain a copy of what is sent into it but baos is also set when NonCloneableHash is used and as a result the archived() method on NonCloneableHash is never called. The error manifests itself at lines 542 to 548 where the code in TLS12HandshakeHash is: public byte[] archived() { if (baos != null) { return baos.toByteArray(); } else { return transcriptHash.archived(); } } So if baos is null the archived() method on transcriptHash is called, but baos is only null if transcriptHash is a CloneableHash and as a result the exception will always be thrown as CloneableHash does not support the archived() method. The setting of baos is swapped and the constructor should actually read: T12HandshakeHash(CipherSuite cipherSuite) { MessageDigest md = JsseJce.getMessageDigest(cipherSuite.hashAlg.name); if (md instanceof Cloneable) { transcriptHash = new CloneableHash(md); this.baos = new ByteArrayOutputStream(); } else { transcriptHash = new NonCloneableHash(md); this.baos = null; } } Further inspection revealed the same issue applies to the constructor of T13HandshakeHash on line 551 - the setting of baos is also swapped there, so the same failure will occur.
22-11-2018
The archived() method is designed to be optional, However, it is not used as an optional operation in the the CertificateVerify. We either update this method, or update the caller to treat it as optional operation. Pallavi, can we ask for more information about the thread stacks that the user run into the problem? As may help us speed up the evaluation of the impact.
20-11-2018