JDK-6314584 : intermittent "RSA PreMasterSecret error" during ssl handshake
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 1.4.2_09
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux_redhat_3.0
  • CPU: x86
  • Submitted: 2005-08-23
  • Updated: 2010-09-30
  • Resolved: 2005-09-02
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
1.4.2_10 b02Fixed
Description
During an SSL handshake, the ssl server (using JSSE from within 
tomcat), fails after receiving the ClientKeyExchange message from 
client.

It receives the encryptedPreMasterSecret from client in the 
ClientKeyExchange message. Then, as it is decrypting the 
preMasterSecret, it complains "RSA PreMasterSecret error, generating 
random secret" in the debug log and returns a handshake_failure to the
client (even though it does display the decrypted preMasterSecret in 
the log). The cipher used here is RSA/EBC/PKCS1Padding.

When decrypting the same received encrypted PreMasterSecret 
with the associated private key, using openssl, it works fine. So, 
it can be assumed that the ssl server did receive a good encrypted 
preMasterSecret, but somehow fails thereafter.

Also, the problem is intermittent in nature as well so not every
exchange fails.

The problem is reproductible with 1.4.2_09 but not with 1.5.

To reproduce the problem :
1. setup an ssl client and server using JSSE 
2. try client server ssl connections repeatedly, using JSSE internal implementation for cipher  RSA/ECB/PKCS1Padding 
3. It fails almost after every 10-15 connections

Debug logs :
*** ClientHello, TLSv1
RandomCookie:  GMT: -1463065024 bytes = { 155, 208, 247, 169, 101, 29,
125, 150, 170, 178, 47, 241, 208, 188, 48, 166, 226, 191, 6, 96, 220, 
200, 254, 56, 98, 28, 10, 138 } Session ID:
{} Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA,
SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,
SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,
SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, 
SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA,
SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,
SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,
SSL_RSA_EXPORT_WITH_RC4_40_MD5] Compression Methods:  { 0 }
***
%% Created:  [Session-154, SSL_RSA_WITH_RC4_128_MD5]
*** ServerHello, TLSv1
RandomCookie:  GMT: 1105647884 bytes = { 124, 251, 153, 219, 131, 109,
41, 218, 178, 128, 136, 230, 118, 217, 15, 76, 176, 140, 138, 254, 49,
85, 147, 150, 43, 55, 95, 226 } Session ID:
 {66, 231, 217, 12, 134, 15, 27, 14, 213, 148, 107, 237, 206, 217, 177,
133, 244, 202, 125, 220, 41, 95, 215, 126, 96, 29, 219, 85, 172, 226, 
191, 24} Cipher Suite:
SSL_RSA_WITH_RC4_128_MD5 Compression Method: 0
***
Cipher suite:  SSL_RSA_WITH_RC4_128_MD5
*** Certificate chain
chain [0] = [
[
  Version: V1
  Subject: CN=xxxx, OU=xxxx, O=xxxx, L=xxxx, ST=xxxx, C=US
  Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4

  Key:  SunJSSE RSA public key:
  public exponent:
    010001
  modulus:
    d1757a28 059e0295 51eaa066 4c816d5d 470fd889 30f3e6d0 a921ba73 fccebca0
    13b023fd 4e7a7a43 924dbe99 c1709fd2 aa4aca8f 72bfd186 c910043e df2d2eb5
    6270dea1 05fbc962 edd7f2ac 395c8b87 74fade29 e61636be aa611104 fbaeddd0
    a5c1a4c3 fe52dd0d 6e43a07d 242e3e36 8cb13abb 9facc5de 2ead6b05 b8527a4d
  Validity: [Wrom: YCGPKYLEJGDGVCJVTLBXFGGMEPYOQ
               To: Tue Jul 25 17:50:09 EDT 2006]
  Issuer: CN=xxxx, OU=xxxx, O=xxxx, L=xxxx, ST=xxxx, C=US
  SerialNumber: [    00]

]
  Algorithm: [MD5withRSA]
  Signature:
0000: 5E EB 98 BF 9B F7 66 EA   08 2D 16 95 88 C3 F6 08  ^.....f..-......
0010: 24 EC 0F 99 37 9A A5 36   E5 F3 49 49 27 F4 95 EF  $...7..6..II'...
0020: 58 CC 26 AF DA DF DF 3E   6C 10 31 23 27 D5 0D FE  X.&....>l.1#'...
0030: F5 F7 DD 23 32 6A DE 3B   72 D3 19 D1 7E 42 36 D6  ...#2j.;r....B6.
0040: F2 93 B8 FF 8E E6 1C BE   87 D3 6F 86 27 B8 B3 BA  ..........o.'...
0050: 00 A2 68 02 8D 4B 42 CF   15 53 A7 18 64 F8 34 94  ..h..KB..S..d.4.
0060: 0D A9 66 F7 17 37 11 69   A8 1D 95 AC D6 CC 58 BA  ..f..7.i......X.
0070: A6 4C D7 3C FA 00 0A 81   E5 1E 66 97 48 3E 97 9F  .L.<......f.H>..

]
***
*** ServerHelloDone
TP-Processor4, WRITE: TLSv1 Handshake, length = 699 TP-Processor4, 
READ: TLSv1 Handshake, length = 133
JsseJCE: Using JSSE internal implementation for cipher 
RSA/ECB/PKCS1Padding RSA PreMasterSecret error, generating random 
secret
*** ClientKeyExchange, RSA PreMasterSecret, TLSv1 Random
Secret:  { 3, 1, 196, 122, 203, 157, 246, 125, 202, 233, 158, 140, 
176, 5, 136, 91, 156, 253, 112, 43, 144, 19, 229, 234, 201, 82, 123, 
66, 2, 123, 26, 167, 74, 224, 35, 141, 162, 85, 20, 216, 155, 102, 
138, 157, 54, 223, 182, 35 } SESSION KEYGEN:
PreMaster Secret:
0000: 03 01 C4 7A CB 9D F6 7D   CA E9 9E 8C B0 05 88 5B  ...z...........[
0010: 9C FD 70 2B 90 13 E5 EA   C9 52 7B 42 02 7B 1A A7  ..p+.....R.B....
0020: 4A E0 23 8D A2 55 14 D8   9B 66 8A 9D 36 DF B6 23  J.#..U...f..6..#
CONNECTION KEYGEN:
Client Nonce:
0000: A9 CB 66 40 9B D0 F7 A9   65 1D 7D 96 AA B2 2F F1  ..f@....e...../.
0010: D0 BC 30 A6 E2 BF 06 60   DC C8 FE 38 62 1C 0A 8A  ..0....`...8b...
Server Nonce:
0000: 42 E7 D9 0C 7C FB 99 DB   83 6D 29 DA B2 80 88 E6  B........m).....
0010: 76 D9 0F 4C B0 8C 8A FE   31 55 93 96 2B 37 5F E2  v..L....1U..+7_.
Master Secret:
0000: F1 42 DF DB BD 42 1E 11   71 AA 74 4B D1 B6 C0 C0  .B...B..q.tK....
0010: 58 9D FF FC 7C AD 04 48   00 0B F9 A5 FB F8 C6 BC  X......H........
0020: 58 DB 61 1D FA FF DB FC   97 D8 FC 4B CF 0F 57 B7  X.a........K..W.
Client MAC write Secret:
0000: 21 7C 6D 5B C9 33 07 AB   4B 94 27 17 A3 5E 74 FA  !.m[.3..K.'..^t.
Server MAC write Secret:
0000: 9B F4 83 98 88 23 66 7A   76 FA E8 90 26 61 FD BD  .....#fzv...&a..
Client write key:
0000: 0F 56 9C 82 DE C6 CA 3E   3F 4D B2 19 31 AD 64 3B  .V.....>?M..1.d;
Server write key:
0000: C7 45 9B 3D 36 7B 5A 23   23 E9 49 0F 36 6F 5E C6  .E.=6.Z##.I.6o^.
... no IV for cipher
TP-Processor4, READ: TLSv1 Change Cipher Spec, length = 1
JsseJCE: Using JSSE internal implementation for cipher RC4 
TP-Processor4, READ: TLSv1 Handshake, length = 32 TP-Processor4, SEND 
TLSv1 ALERT:  fatal, description = handshake_failure TP-Processor4, 
WRITE: TLSv1 Alert, length = 2 TP-Processor4, called closeSocket()

Comments
EVALUATION The problem is that the encrypted premaster secret has the highest byte set to zero. The client (OpenSSL?) apparently chops off that byte before sending, i.e. it sends 127 instead of 128 bytes. The JSafe RSA we are using rejects that. The fix would have to pad the premaster secret to the length of the RSA modulus used. This is not an issue in 1.5 and beyond because we do not use the JSafe code any more.
23-08-2005

WORK AROUND Use a 3rd party JCE crypto provider for RSA.
23-08-2005