Duplicate :
|
FULL PRODUCT VERSION : java version " 1.7.0_21 " Java(TM) SE Runtime Environment (build 1.7.0_21-b12) Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode) ADDITIONAL OS VERSION INFORMATION : I can reproduce this on my mac: Darwin bebop.local 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64 My users have seen this on Ubuntu 12.04.2 LTS with JDK 7u15-2.3.7-0ubuntu1~12.04.1 My users have also seen this on RHEL 6.4 with java-1.7.0-oracle-1.7.0.17-1jpp.1.el6_4.x86_64 A DESCRIPTION OF THE PROBLEM : The best description is in the following forum thread: https://forums.oracle.com/forums/thread.jspa?messageID=10999587 When running an SSL server on the JVM using any JDK beyond 1.7u5, clients will eventually (around 5% of the time in my testing) receive a handshake error due to a problem with handling padding in the Diffie Hellman ciphers. I've included precise steps to reproduce the issue along with this ticket. The problem goes away completely if one chooses non-diffie-hellman cipher suites. This, however, doesn't solve the problem for people that require DHE for business reasons. ActiveMQ Apollo has run into the same issue: https://issues.apache.org/jira/browse/APLO-287 As has PuppetDB: http://projects.puppetlabs.com/issues/19884 REGRESSION. Last worked in version 6u45 STEPS TO FOLLOW TO REPRODUCE THE PROBLEM : Here is a setup that replicates the problem. Create a keypair in a java keystore, and run this server: package edu.berkeley.ist.scratch; import javax.net.ssl.SSLServerSocket; import javax.net.ssl.SSLServerSocketFactory; import javax.net.ssl.SSLSocket; import java.io.BufferedReader; import java.io.InputStream; import java.io.InputStreamReader; public class EchoSSLServer { public static void main(String[] arstring) { try { SSLServerSocketFactory sslserversocketfactory = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault(); SSLServerSocket sslserversocket = (SSLServerSocket) sslserversocketfactory.createServerSocket(9999); while (true) { SSLSocket sslsocket = (SSLSocket) sslserversocket.accept(); InputStream inputstream = sslsocket.getInputStream(); InputStreamReader inputstreamreader = new InputStreamReader(inputstream); BufferedReader bufferedreader = new BufferedReader(inputstreamreader); String string = null; while ((string = bufferedreader.readLine()) != null) { System.out.println(string); System.out.flush(); } } } catch (Exception exception) { exception.printStackTrace(); } } } Then run this bash program; it will repeatedly connect to the above server: count=0 while [ 1 -eq 1 ] do echo " Hello " | openssl s_client -connect localhost:9999 >/dev/null if [ $? -ne 0 ]; then exit 1 fi ((count+=1)) echo $count done After a few hundered iterations, you will see something like this on the client side: failed connect or ssl handshake: ,IO::Socket::INET6 configuration failederror:00000000:lib(0):func(0):reason(0) at ssl-client.pl line 6. EXPECTED VERSUS ACTUAL BEHAVIOR : EXPECTED - I'd expect zero handshake errors. ACTUAL - After a few hundered iterations, you will see something like this on the client side: failed connect or ssl handshake: ,IO::Socket::INET6 configuration failederror:00000000:lib(0):func(0):reason(0) at ssl-client.pl line 6. So a certain percentage of the time, clients will not be able to connect to the JVM SSL socket. ERROR MESSAGES/STACK TRACES THAT OCCUR : Given the above code to reproduce the problem, you see the following error on the JVM side: And this on the server side: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 81 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1886) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:974) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:882) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:154) at java.io.BufferedReader.readLine(BufferedReader.java:317) at java.io.BufferedReader.readLine(BufferedReader.java:382) at edu.berkeley.ist.scratch.EchoSSLServer.main(EchoSSLServer.java:67) Caused by: javax.crypto.BadPaddingException: Invalid Padding length: 81 at sun.security.ssl.CipherBox.removePadding(CipherBox.java:684) at sun.security.ssl.CipherBox.decrypt(CipherBox.java:423) at sun.security.ssl.InputRecord.decrypt(InputRecord.java:154) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:969) ... 11 more After enabling debug logging for SSL handshake, I see this: main, READ: SSL v2, contentType = Handshake, translated length = 89 *** ClientHello, TLSv1 RandomCookie: GMT: 0 bytes = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 93, 75, 180, 53, 239, 108, 51, 69, 35, 182, 61, 245, 71, 68, 28, 211 } Session ID: {} Cipher Suites: [TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_SEED_CBC_SHA, TLS_DHE_DSS_WITH_SEED_CBC_SHA, TLS_RSA_WITH_SEED_CBC_SHA, SSL_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_MD5, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5, SSL_RSA_EXPORT_WITH_RC4_40_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] Compression Methods: { 0 } *** %% Initialized: [Session-232, SSL_NULL_WITH_NULL_NULL] %% Negotiating: [Session-232, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA] *** ServerHello, TLSv1 RandomCookie: GMT: 1361908238 bytes = { 210, 137, 20, 192, 173, 40, 205, 3, 242, 185, 80, 19, 198, 183, 44, 63, 3, 149, 97, 175, 153, 239, 243, 97, 92, 200, 110, 212 } Session ID: {81, 45, 18, 14, 137, 154, 98, 252, 20, 124, 81, 4, 214, 8, 231, 121, 175, 133, 142, 252, 20, 12, 99, 201, 24, 9, 83, 15, 239, 34, 39, 61} Cipher Suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA Compression Method: 0 Extension renegotiation_info, renegotiated_connection: <empty> *** Cipher suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA *** Certificate chain chain [0] = [ [ Version: V3 Subject: CN=localhost, OU=IST, O= " University of California, Berkeley " , L=Berkeley, ST=CA, C=US Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11 Key: Sun RSA public key, 1024 bits modulus: 100318692587772653487949066254487989918157604545853798061718708748206860158041498924763974217777890645501700365160396601829010852095657193155138078345180085242824268946142257050055468216742995753689139662471328635164222926500994444770172537176817688500963591823961418447303232823489913199105654849046538472289 public exponent: 65537 Validity: [From: Mon Feb 25 11:43:44 PST 2013, To: Thu Feb 20 11:43:44 PST 2014] Issuer: CN=localhost, OU=IST, O= " University of California, Berkeley " , L=Berkeley, ST=CA, C=US SerialNumber: [ 2cba5851] Certificate Extensions: 1 [1]: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: EC 0B B0 6A 3F 7E D1 9A FD 97 E7 DD C5 7C 00 1E ...j?........... 0010: BD 35 5D 31 .5]1 ] ] ] Algorithm: [SHA256withRSA] Signature: 0000: 4D C2 29 84 58 2B D6 D2 47 D9 A9 45 68 54 80 9C M.).X+..G..EhT.. 0010: 9F CD B1 4D FD 9E B6 AC 74 DF F8 B2 B6 16 BB 9C ...M....t....... 0020: 55 F9 60 DF 3E 86 9E AE 84 65 DC 59 86 92 84 EC U.`.>....e.Y.... 0030: F7 7F D2 EE 39 53 26 87 3F B8 69 9E 58 20 C5 75 ....9S&.?.i.X .u 0040: A8 38 23 78 85 2D 61 F8 19 15 8F 5F C6 BF 08 6C .8#x.-a...._...l 0050: DA AD F5 C3 B2 84 67 B3 D4 A2 3C 88 D8 B4 15 15 ......g...<..... 0060: 10 10 B5 D1 8F 93 2F F4 62 D8 A2 24 74 C2 27 1F ....../.b..$t.'. 0070: 33 E4 1C 6A 0C 61 1F DB DC E5 4C F3 B0 83 46 CB 3..j.a....L...F. ] *** *** Diffie-Hellman ServerKeyExchange DH Modulus: { 233, 230, 66, 89, 157, 53, 95, 55, 201, 127, 253, 53, 103, 18, 11, 142, 37, 201, 205, 67, 233, 39, 179, 169, 103, 15, 190, 197, 216, 144, 20, 25, 34, 210, 195, 179, 173, 36, 128, 9, 55, 153, 134, 157, 30, 132, 106, 171, 73, 250, 176, 173, 38, 210, 206, 106, 34, 33, 157, 71, 11, 206, 125, 119, 125, 74, 33, 251, 233, 194, 112, 181, 127, 96, 112, 2, 243, 206, 248, 57, 54, 148, 207, 69, 238, 54, 136, 193, 26, 140, 86, 171, 18, 122, 61, 175 } DH Base: { 48, 71, 10, 213, 160, 5, 251, 20, 206, 45, 157, 205, 135, 227, 139, 199, 209, 177, 197, 250, 203, 174, 203, 233, 95, 25, 10, 167, 163, 29, 35, 196, 219, 188, 190, 6, 23, 69, 68, 64, 26, 91, 44, 2, 9, 101, 216, 194, 189, 33, 113, 211, 102, 132, 69, 119, 31, 116, 186, 8, 77, 32, 41, 216, 60, 28, 21, 133, 71, 243, 169, 241, 162, 113, 91, 226, 61, 81, 174, 77, 62, 90, 31, 106, 112, 100, 243, 22, 147, 58, 52, 109, 63, 82, 146, 82 } Server DH Public Key: { 172, 67, 73, 181, 25, 206, 209, 77, 79, 31, 226, 46, 83, 69, 140, 140, 22, 195, 111, 42, 172, 99, 197, 235, 173, 157, 97, 136, 79, 75, 116, 206, 222, 83, 158, 57, 134, 161, 188, 133, 1, 207, 15, 220, 98, 36, 187, 164, 222, 202, 244, 16, 77, 55, 212, 165, 120, 146, 239, 61, 5, 118, 28, 83, 254, 68, 1, 221, 73, 103, 218, 108, 174, 252, 122, 71, 71, 31, 195, 106, 87, 23, 53, 162, 130, 82, 44, 61, 9, 149, 25, 50, 93, 91, 9, 76 } Signed with a DSA or RSA public key *** ServerHelloDone main, WRITE: TLSv1 Handshake, length = 1185 main, READ: TLSv1 Handshake, length = 102 *** ClientKeyExchange, DH DH Public key: { 40, 45, 195, 251, 225, 147, 146, 211, 158, 138, 201, 109, 148, 41, 22, 10, 146, 233, 14, 87, 55, 145, 189, 247, 21, 113, 123, 26, 198, 8, 225, 169, 164, 202, 107, 59, 148, 133, 111, 72, 128, 82, 20, 47, 223, 39, 108, 44, 36, 17, 31, 170, 126, 254, 201, 192, 233, 192, 67, 155, 77, 194, 190, 155, 70, 169, 219, 48, 55, 236, 150, 214, 255, 183, 255, 87, 253, 228, 7, 231, 211, 72, 14, 159, 156, 34, 22, 133, 64, 21, 163, 97, 80, 3, 67, 165 } SESSION KEYGEN: PreMaster Secret: 0000: 00 90 1E 38 9D 9E 79 DD 7B 1F B5 0C 7A 58 82 38 ...8..y.....zX.8 0010: 8E 19 8A CD A0 A3 EF A1 DC 3B B1 3E FC 35 C8 97 .........;.>.5.. 0020: 1E AF 62 4A 7C 95 52 6B A0 8E A6 94 25 D1 20 06 ..bJ..Rk....%. . 0030: 77 6B 7A 19 7A C7 D8 12 DF 61 97 5E 8E 10 40 E2 wkz.z....a.^..@. 0040: 3B FA A5 64 9B EA D8 50 9C 35 84 36 31 12 04 B2 ;..d...P.5.61... 0050: 4D CE 33 BB 9D E1 8A 2C 99 38 13 99 78 75 8F BF M.3....,.8..xu.. CONNECTION KEYGEN: Client Nonce: 0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0010: 5D 4B B4 35 EF 6C 33 45 23 B6 3D F5 47 44 1C D3 ]K.5.l3E#.=.GD.. Server Nonce: 0000: 51 2D 12 0E D2 89 14 C0 AD 28 CD 03 F2 B9 50 13 Q-.......(....P. 0010: C6 B7 2C 3F 03 95 61 AF 99 EF F3 61 5C C8 6E D4 ..,?..a....a\.n. Master Secret: 0000: 6D 72 A2 A0 98 23 31 53 0F 02 E9 8A 95 8A 6E B2 mr...#1S......n. 0010: 4E 61 4F 04 37 16 4C FB D5 3C DC 89 38 5A 30 37 NaO.7.L..<..8Z07 0020: AB E3 E0 4D 36 01 68 1B C1 D6 33 A7 C0 CB FD 1C ...M6.h...3..... Client MAC write Secret: 0000: 38 45 A5 8C BA 1D 04 9E 19 21 3F C1 2F 1C A0 82 8E.......!?./... 0010: 93 8E 89 88 .... Server MAC write Secret: 0000: C8 26 BC FA 05 EC DA 66 56 98 31 87 79 87 F3 37 .&.....fV.1.y..7 0010: CC 7B 09 A3 .... Client write key: 0000: EA E5 A4 37 30 55 20 61 14 51 09 DF 50 EC 8A 94 ...70U a.Q..P... 0010: E4 4A BD 84 0B D4 7E 9D .J...... Server write key: 0000: A3 5C C0 D0 03 49 8E BA 1C 62 88 BF 98 61 10 CC .\...I...b...a.. 0010: 57 E6 72 BB 3B 88 9F 3E W.r.;..> Client write IV: 0000: 64 A7 95 AA F7 61 82 7B d....a.. Server write IV: 0000: A8 62 ED B7 C9 8C 1D 76 .b.....v main, READ: TLSv1 Change Cipher Spec, length = 1 main, READ: TLSv1 Handshake, length = 40 %% Invalidated: [Session-232, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA] main, SEND TLSv1 ALERT: fatal, description = handshake_failure main, WRITE: TLSv1 Alert, length = 2 main, called closeSocket() main, handling exception: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 81 javax.net.ssl.SSLHandshakeException: Invalid Padding length: 81 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1886) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:974) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:882) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:154) at java.io.BufferedReader.readLine(BufferedReader.java:317) at java.io.BufferedReader.readLine(BufferedReader.java:382) at edu.berkeley.ist.scratch.EchoSSLServer.main(EchoSSLServer.java:67) Caused by: javax.crypto.BadPaddingException: Invalid Padding length: 81 at sun.security.ssl.CipherBox.removePadding(CipherBox.java:684) at sun.security.ssl.CipherBox.decrypt(CipherBox.java:423) at sun.security.ssl.InputRecord.decrypt(InputRecord.java:154) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:969) ... 11 more REPRODUCIBILITY : This bug can be reproduced always. ---------- BEGIN SOURCE ---------- package ssl.bug; import javax.net.ssl.SSLServerSocket; import javax.net.ssl.SSLServerSocketFactory; import javax.net.ssl.SSLSocket; import java.io.BufferedReader; import java.io.InputStream; import java.io.InputStreamReader; public class EchoSSLServer { public static void main(String[] arstring) { try { SSLServerSocketFactory sslserversocketfactory = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault(); SSLServerSocket sslserversocket = (SSLServerSocket) sslserversocketfactory.createServerSocket(9999); while (true) { SSLSocket sslsocket = (SSLSocket) sslserversocket.accept(); InputStream inputstream = sslsocket.getInputStream(); InputStreamReader inputstreamreader = new InputStreamReader(inputstream); BufferedReader bufferedreader = new BufferedReader(inputstreamreader); String string = null; while ((string = bufferedreader.readLine()) != null) { System.out.println(string); System.out.flush(); } } } catch (Exception exception) { exception.printStackTrace(); } } } Then run this bash program; it will repeatedly connect to the above server and will show occasional handshake errors: count=0 while [ 1 -eq 1 ] do echo " Hello " | openssl s_client -connect localhost:9999 >/dev/null if [ $? -ne 0 ]; then exit 1 fi ((count+=1)) echo $count done ---------- END SOURCE ---------- CUSTOMER SUBMITTED WORKAROUND : Disable Diffie-Hellman based cipher suites. This doesn't work in environments where DHE is required for security/certificate reasons.