JDK-7103725 : REGRESSION - 6u29 breaks ssl connectivity using TLS_DH_anon_WITH_AES_128_CBC_SHA
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 6u26,6u29
  • Priority: P1
  • Status: Resolved
  • Resolution: Fixed
  • OS:
    solaris_8,windows_xp,windows_2008,windows_7 solaris_8,windows_xp,windows_2008,windows_7
  • CPU: x86
  • Submitted: 2011-10-21
  • Updated: 2012-05-25
  • Resolved: 2011-11-09
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other JDK 6 Other
1.4.2_35,OpenJDK6Fixed 6u30 b12Fixed OpenJDK6Fixed
Related Reports
Duplicate :  
Relates :  
Description
FULL PRODUCT VERSION :
On the server:

Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)

On the client:

Java versino "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) 64-bit Server VM (build 20.4-b02, mixed-mode)

ADDITIONAL OS VERSION INFORMATION :
On the server:
# uname -a
Linux squid 2.6.26-2-amd64 #1 SMP Mon Jun 13 16:29:33 UTC 2011 x86_64 GNU/Linux

On the client: Windows 7 Ultimate

A DESCRIPTION OF THE PROBLEM :
System was working fine with server and client on Java 6 versions before 6u29 (e.g., 6u27). The server and client didn't need to be java version lock-stepped either.

A customer upgraded their client Windows 7 PC to 6u29 and communications between the client and server immediately fails. I was able to duplicate this field issue in our development lab by updating a client to 6u29.

Client and server currently communicate via a SSL socket using:

TLS_DH_anon_WITH_AES_128_CBC_SHA

If I change the SSL socket to use:

SSL_DH_anon_WITH_RC4_128_MD5

Communications between client and server work again. But I have hundreds of customers and to change to this much less secure specification is not viable.

Upgrading the server to 6u29 also doesn't help the situation. With client and server both on 6u29, failures still happen.

This breakage may be related to the work done in 6u29 for CVE-2011-3389.

REGRESSION.  Last worked in version 6u26

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Establish a SSL connection using the TLS_DH_anon_WITH_AES_128_CBC_SHA specification and it will fail after a small number of message exchanges. In our case, a server write, a client write, a server write followed by a consistently failing client write. All writes are externalized Java String or Long objects.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
They work
ACTUAL -
Exceptions:

On the server:

java.net.SocketException: Socket closed
java.net.SocketException: Socket closed
  at java.net.PlainSocketImpl.socketAccept(Native Method)
  at java.net.PlainSocketImpl.accept(Unknown Source)
  at java.net.ServerSocket.implAccept(Unknown Source)
  at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(Unknown Source)
  at com.example.package.ClientConnectionInterface$ServerSocketHandler.run(ClientConnectionInterface.java:681)

And on the client:

javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkWrite(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppOutputStream.write(Unknown Source)
	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(Unknown Source)
	at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)
	at java.io.ObjectOutputStream.writeFatalException(Unknown Source)
	at java.io.ObjectOutputStream.writeObject(Unknown Source)
	at com.example.package.PrioritySocketManager.initialize(PrioritySocketManager.java:408)
	at com.example.package.connectToSPM(SCAI.java:1224)
	at com.example.package.connect(SCAI.java:533)
	at com.example.package.ConnectionAdapter.establishConnection(ConnectionAdapter.java:283)
	at com.example.package.ConnectionHandler.fullLogin(ConnectionHandler.java:446)
	at com.example.package.ConnectionHandler.access$1200(ConnectionHandler.java:67)
	at com.example.package.ConnectionHandler$3.run(ConnectionHandler.java:299)
Caused by: javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException
	at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppOutputStream.write(Unknown Source)
	at java.io.ObjectOutputStream$BlockDataOutputStream.write(Unknown Source)
	at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source)
	at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
	at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
	at java.io.ObjectOutputStream.writeObject0(Unknown Source)
	... 8 more
Caused by: java.lang.IndexOutOfBoundsException
	at java.io.ByteArrayOutputStream.write(Unknown Source)
	... 14 more

ERROR MESSAGES/STACK TRACES THAT OCCUR :
See exceptions above

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Change the SSL specification to SSL_DH_anon_WITH_RC4_128_MD5 (others may also work).

Comments
EVALUATION The more obvious impact of this bug is to the MS JDBC Driver and MS SQL Server. Please see 7105007. I am closing that bug as a duplicate of this one. We have confirmed 7105007 is a duplicate of this, but will be including more info in that bug as external customers will be finding that bug via searches.
16-11-2011

EVALUATION Updated attachment.
26-10-2011

EVALUATION This regression affects: 6u29, 5.0u32, and 1.4.2_35. Will need sustaining to determine appropriate release vehicles. It does *NOT* affect 7u1, as the AppOutputStream.write() call returns immediately if len==0.
26-10-2011

EVALUATION This was caused by 7064341. It's legal to call: AppOutputStream.write(b, 0, 0) // b is empty array (byte b [0]) We will drive the machinery of the handshake, but will not actually write any data into the OutputRecord r. However, when you do this on the first record of the payload, you go through this code: if (isFirstRecordOfThePayload && c.needToSplitPayload()) { howmuch = Math.min(0x01, r.availableDataBytes()); which sets howmuch to 1. Then later: if (howmuch > 0) { r.write(b, 0, 1); rightly throws an java.lang.IndexOutOfBoundsException since b is empty (zero byte array). You need to set howmuch to 0 if len == 0, otherwise min(0x1, r.availableDataBytes);
26-10-2011