JDK-8161086 : DTLS handshaking fails if some messages were lost
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 9
  • Priority: P2
  • Status: Closed
  • Resolution: Delivered
  • Submitted: 2016-07-08
  • Updated: 2016-11-29
  • Resolved: 2016-11-29
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 9
9Resolved
Related Reports
Blocks :  
JDK-8159416 - javax/net/ssl/DTLS/CipherSuite.java failed on timeout
Duplicate :  
JDK-8167680 - DTLS implementation bugs
Duplicate :  
JDK-8163419 - Final CCS and Finished DTLS messages can't be re-transmitted
Relates :  
JDK-8167680 - DTLS implementation bugs
Description
DTLS client usually sends three messages in flight #5:

    ClientKeyExchange                                          
    ChangeCipherSpec                                        
    Finished

See https://tools.ietf.org/html/rfc4347 for details.

DTLS can be used over UDP connection which may result to packets lost. If a ClientKeyExchange message was lost while DTLS handshaking then it results to "ChangeCipherSpec message sequence violation" error because server tries to handle ChangeCipherSpec:

javax.net.ssl.SSLProtocolException: ChangeCipherSpec message sequence violation
	at sun.security.ssl.HandshakeStateManager.changeCipherSpec(java.base@9-internal/HandshakeStateManager.java:891)
	at sun.security.ssl.Handshaker.receiveChangeCipherSpec(java.base@9-internal/Handshaker.java:1137)
	at sun.security.ssl.SSLEngineImpl.processInputRecord(java.base@9-internal/SSLEngineImpl.java:1142)
	at sun.security.ssl.SSLEngineImpl.readRecord(java.base@9-internal/SSLEngineImpl.java:998)
	at sun.security.ssl.SSLEngineImpl.readNetRecord(java.base@9-internal/SSLEngineImpl.java:895)
	at sun.security.ssl.SSLEngineImpl.unwrap(java.base@9-internal/SSLEngineImpl.java:673)
	at javax.net.ssl.SSLEngine.unwrap(java.base@9-internal/SSLEngine.java:624)
	at DTLSOverDatagram.handshake(DTLSOverDatagram.java:255)
	at DTLSOverDatagram.doServerSide(DTLSOverDatagram.java:116)
	at DTLSOverDatagram$Server.run(DTLSOverDatagram.java:678)
	at java.lang.Thread.run(java.base@9-internal/Thread.java:843)

It might be better if a server handshaker could recognize such a situation, and let client re-send missing packets, so that handshaking may be successfully finished.
Comments
The issue will be fixed within JDK-8167680.
13-10-2016
Here is an updated patch: http://cr.openjdk.java.net/~asmotrak/8161086/webrev.draft.01/ After applying this patch, DTLS handshaker seems to be able to finish handshaking if mandatory messages (like ClientHello, ServerHello, CCS, etc) were lost. But it still fails if Certificate message was lost, and non-anon cipher suite is used (in this case, Certificate message is required).
16-07-2016
There is one more problem. DTLS server usually sends the following messages in flight #4: ServerHello Certificate ServerHelloDone DTLS client may also fail if Certificate message is lost. Here is a piece of log: ... Server thread, WRITE: DTLSv1.2 Handshake, ServerHello, length = 89, epoch = 0, sequence number = 1 Server thread, WRITE: DTLSv1.2 Handshake, Certificate, length = 865, epoch = 0, sequence number = 2 Server thread, WRITE: DTLSv1.2 Handshake, ServerHelloDone, length = 12, epoch = 0, sequence number = 3 Server: handshake(): need wrap: send 3 packets CustomDatagramSocketImpl: send() called, length = 102 CustomDatagramSocketImpl: lose a packet CustomDatagramSocketImpl: send() called, length = 878 CustomDatagramSocketImpl: send() called, length = 25 Server: Need DTLS records, handshake status is NEED_UNWRAP Server: handshake(): wait for a packet CustomDatagramSocketImpl: receive() called Client: handshake(): received a packet, length = 878 Client thread, READ: DTLSv1.2 Handshake, length = 865, epoch = 0, sequence number = 2 check handshake state: certificate[11] Client: Need DTLS records, handshake status is NEED_UNWRAP Client: handshake(): wait for a packet CustomDatagramSocketImpl: receive() called Client: handshake(): received a packet, length = 25 Client thread, fatal error: 10: Handshake message sequence violation, 11 javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 11 Unexpected exception on client side: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 11 at sun.security.ssl.Handshaker.checkThrown(java.base@9-internal/Handshaker.java:1489) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(java.base@9-internal/SSLEngineImpl.java:490) at sun.security.ssl.SSLEngineImpl.readNetRecord(java.base@9-internal/SSLEngineImpl.java:738) at sun.security.ssl.SSLEngineImpl.unwrap(java.base@9-internal/SSLEngineImpl.java:673) at javax.net.ssl.SSLEngine.unwrap(java.base@9-internal/SSLEngine.java:624) at DTLSOverDatagram.handshake(DTLSOverDatagram.java:255) at DTLSOverDatagram.doClientSide(DTLSOverDatagram.java:148) at DTLSOverDatagram$Client.run(DTLSOverDatagram.java:722) at java.lang.Thread.run(java.base@9-internal/Thread.java:843) Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 11 at sun.security.ssl.HandshakeStateManager.check(java.base@9-internal/HandshakeStateManager.java:393) at sun.security.ssl.ClientHandshaker.processMessage(java.base@9-internal/ClientHandshaker.java:211) at sun.security.ssl.Handshaker.processLoop(java.base@9-internal/Handshaker.java:1003) at sun.security.ssl.Handshaker$1.run(java.base@9-internal/Handshaker.java:942) at sun.security.ssl.Handshaker$1.run(java.base@9-internal/Handshaker.java:939) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(java.base@9-internal/Handshaker.java:1418) at DTLSOverDatagram.runDelegatedTasks(DTLSOverDatagram.java:565) at DTLSOverDatagram.handshake(DTLSOverDatagram.java:304) ... 3 more ... The failure can be reproduced with javax/net/ssl/DTLS/CipherSuite.java test after applying the following patch: http://cr.openjdk.java.net/~asmotrak/8161086/webrev.draft.00/test/javax/net/ssl/DTLS/DTLSOverDatagram.java.patch The test should be run with TLS_RSA_WITH_AES_128_CBC_SHA256 cipher suite and "-Dseed=-4342715767083144063" option which specifies a seed for pseudo-random generator.
12-07-2016
This issue was found while fixing JDK-8159416 which updates DTLS tests to use a custom DatagramSocketImpl which simulates packet lost. https://bugs.openjdk.java.net/browse/JDK-8159416?focusedCommentId=13965637&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13965637 The failure can be reproduced with javax/net/ssl/DTLS/CipherSuite.java test after applying the following patch: http://cr.openjdk.java.net/~asmotrak/8159416/webrev.01/ The test should be run with TLS_RSA_WITH_AES_128_CBC_SHA256 cipher suite and "-Dseed=7626304887466190664" option which specifies a seed for pseudo-random generator.
12-07-2016
Here is a draft for this bug and JDK-8159416 http://cr.openjdk.java.net/~asmotrak/8161086/webrev.draft.00/ After applying this patch, DTLS handshaker seems to be able to finish handshaking if a CCS message was lost. But I have not tested this patch well yet. And it still fails if Certificate message was lost (see above).
12-07-2016
The patch above probably is not good enough since doesn't take into account that packets may be re-ordered, and ClientKeyExchange may arrive right after ChangeCipherSpec.
08-07-2016
This issue may be fixed with the following patch which updates DTLSInputRecord to reject CCS message with unexpected record number. This would require a remote peer to re-send packets. diff -r a1d07c140e66 src/java.base/share/classes/sun/security/ssl/DTLSInputRecord.java --- a/src/java.base/share/classes/sun/security/ssl/DTLSInputRecord.java Fri Jul 08 10:41:12 2016 -0700 +++ b/src/java.base/share/classes/sun/security/ssl/DTLSInputRecord.java Fri Jul 08 13:31:45 2016 -0700 @@ -1,5 +1,5 @@ /* - * Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2015, 2016, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -32,7 +32,6 @@ import javax.net.ssl.*; -import sun.security.util.HexDumpEncoder; import static sun.security.ssl.HandshakeMessage.*; /** @@ -52,6 +51,9 @@ Authenticator prevReadAuthenticator; CipherBox prevReadCipher; + // sequence number of last received record + long lastRecordSeq; + DTLSInputRecord() { this.readEpoch = 0; this.readAuthenticator = new MAC(true); @@ -59,6 +61,8 @@ this.prevReadEpoch = 0; this.prevReadCipher = CipherBox.NULL; this.prevReadAuthenticator = new MAC(true); + + lastRecordSeq = -1; } @Override @@ -153,6 +157,12 @@ int contentLen = ((packet.get() & 0xFF) << 8) | (packet.get() & 0xFF); // pos: 11, 12 + boolean unexpected = false; + if (lastRecordSeq >= 0 && recordSeq != lastRecordSeq + 1) { + unexpected = true; + } + lastRecordSeq = recordSeq; + if (debug != null && Debug.isOn("record")) { System.out.println(Thread.currentThread().getName() + ", READ: " + @@ -240,6 +250,12 @@ return null; } + if (contentType == Record.ct_change_cipher_spec && unexpected) { + // reject an unexpected CCS message, + // looks like some handshake messages were lost + return null; + } + reassembler.queueUpFragment( new RecordFragment(plaintextFragment, contentType, majorVersion, minorVersion,
08-07-2016