JDK-6998053 : javax.net.ssl.SSLException: Received fatal alert: illegal_parameter
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 6u21
  • Priority: P3
  • Status: Resolved
  • Resolution: Won't Fix
  • OS: solaris_10
  • CPU: sparc
  • Submitted: 2010-11-06
  • Updated: 2020-10-28
  • Resolved: 2020-10-28
Description
Problem statement:

Customer ran into an issue with SSL. In PROD,the SSL handshake is successful in first attempt. Cu then tries to reuse this cached session and then its fails with this error:

         javax.net.ssl.SSLException: Received fatal alert: illegal_parameter


Here's the SSL debug output leading to the error:

Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
***
[write] MD5 and SHA1 hashes:  len = 191
0000: 01 00 00 BB 03 01 4C CF   71 10 77 DB D9 0D 81 4A  ......L.q.w....J
0010: 1B B7 BF 3B 51 09 24 89   AA F0 30 5D A7 BB B5 C6  ...;Q.$...0]....
0020: 80 30 74 E7 70 80 10 65   B5 D9 60 64 6B D2 B7 1A  .0t.p..e..`dk...
0030: C6 4A EB 3C 92 5C 3D 00   44 00 04 00 05 00 2F 00  .J.<.\=.D...../.
0040: 35 C0 02 C0 04 C0 05 C0   0C C0 0E C0 0F C0 07 C0  5...............
0050: 09 C0 0A C0 11 C0 13 C0   14 00 33 00 39 00 32 00  ..........3.9.2.
0060: 38 00 0A C0 03 C0 0D C0   08 C0 12 00 16 00 13 00  8...............
0070: 09 00 15 00 12 00 03 00   08 00 14 00 11 01 00 00  ................
0080: 3E 00 0A 00 34 00 32 00   17 00 01 00 03 00 13 00  >...4.2.........
0090: 15 00 06 00 07 00 09 00   0A 00 18 00 0B 00 0C 00  ................
00A0: 19 00 0D 00 0E 00 0F 00   10 00 11 00 02 00 12 00  ................
00B0: 04 00 05 00 14 00 08 00   16 00 0B 00 02 01 00     ...............
JobCourier38, WRITE: TLSv1 Handshake, length = 191
[Raw write]: length = 196
[Raw write]: length = 196
0000: 16 03 01 00 BF 01 00 00   BB 03 01 4C CF 71 10 77  ...........L.q.w
0010: DB D9 0D 81 4A 1B B7 BF   3B 51 09 24 89 AA F0 30  ....J...;Q.$...0
0020: 5D A7 BB B5 C6 80 30 74   E7 70 80 10 65 B5 D9 60  ].....0t.p..e..`
0030: 64 6B D2 B7 1A C6 4A EB   3C 92 5C 3D 00 44 00 04  dk....J.<.\=.D..
0040: 00 05 00 2F 00 35 C0 02   C0 04 C0 05 C0 0C C0 0E  .../.5..........
0050: C0 0F C0 07 C0 09 C0 0A   C0 11 C0 13 C0 14 00 33  ...............3
0060: 00 39 00 32 00 38 00 0A   C0 03 C0 0D C0 08 C0 12  .9.2.8..........
0070: 00 16 00 13 00 09 00 15   00 12 00 03 00 08 00 14  ................
0080: 00 11 01 00 00 3E 00 0A   00 34 00 32 00 17 00 01  .....>...4.2....
0090: 00 03 00 13 00 15 00 06   00 07 00 09 00 0A 00 18  ................
00A0: 00 0B 00 0C 00 19 00 0D   00 0E 00 0F 00 10 00 11  ................
00B0: 00 02 00 12 00 04 00 05   00 14 00 08 00 16 00 0B  ................
00C0: 00 02 01 00                                        ....
[Raw read]: length = 5
0000: 15 03 01 00 02                                     .....
[Raw read]: length = 2
0000: 02 2F                                              ./
JobCourier38, READ: TLSv1 Alert, length = 2
JobCourier38, RECV TLSv1 ALERT:  fatal, illegal_parameter
JobCourier38, called closeSocket()
JobCourier38, handling exception: javax.net.ssl.SSLException: Received fatal alert: illegal_parameter


The problem doesn't happen in QA. 

From ssl debug output, one difference is that PROD output contains Elliptical Curve Cryptography (not necessary the cause)

Here's the difference in SSL debug output (in PROD):

Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1} 
Extension ec_point_formats, formats: [uncompressed] 
*** 

Found this on the web:

error:  " fatal, description = illegal_parameter"

Article:
http://forums.sun.com/thread.jspa?threadID=306461

Potential resolution:
	
I was getting illegal_parameter in SSL Handshake with weblogic 6.1. To fix it, I made the com.sun.net.ssl.internal.ssl.Provider be provider 2. I originally had it as provider 3 (preceded by com.sun.rsajca.Provider)

My providers (as listed in jdk131/jre/lib/security/java.security are:

#
# List of providers and their preference orders (see above):
#
security.provider.1=sun.security.provider.Sun
security.provider.2=com.sun.net.ssl.internal.ssl.Provider
security.provider.3=com.sun.rsajca.Provider
security.provider.4=com.sun.crypto.provider.SunJCE

Once I made this change, my handshake negotiation worked just fine.

I also had to add the 3 jsse jars to the classpath BEFORE the weblogic jars.


In the meantime, data collected includes:

logon to cores2-da-sparc-2-b.central
/cores_data/local/bin/acl grant 73800616
cd /cores/73800616

1. SSL debug output -Djavax.net.ssl.debug=false  from 
   good (QA) - APAC/uat_works-d1csi1m5.log
   bad (PROD) system  - p12csi1m1.log

2. explorer output from good and bad system
   good (QA) - QA_nsqeap12/explorer.83d565a3.nsqeap12-2010.11.04.08.10
   bad (PROD) - PROD_dspcsi16/explorer.842b2dae.dspcsi16-2010.10.31.06.10


Old bus with similiar error:
4119461 SSL Connections to Oracle Web Application Server do not work
Comments
This was a long time ago and filed against the old JSSE code. It has not been observed in the new code in JDK's 8/11/15/16, so closing as Will Not Fix for now. Please reopen if this remains an issue in the JDK 7 code. Don't think anyone has asked about this years.
28-10-2020
EVALUATION Closing as a duplicate of 6976121 as described in the note 2 above.
10-02-2011
EVALUATION In wrapping up what I remember about this bug, the server we were connecting to was old and did not recognize the ECC extensions, and really needs to be updated. If SSLv2Hello was disabled on the first connection, connections would not be possible at all, as the server would choke on the first handshake when receiving the ECC extensions. One workaround is to turn off ECC ciphersuites using the system property, which removes the ECC extensions from being sent during any handshakes, or by removing them from the enabled cipher suites. On a renegotation (that is, doing a secondary handshake while being protected by the initial session), we should not be sending things using a SSLv2Hello. SSLv2Hellos are only for the initial handshake on a new connection. On a resumption (that is, a new connection, but trying to resume an existing session), SSLv2Hellos are currently not being sent. We will be fixing 6976121 to include this issue.
10-02-2011
EVALUATION First off, from the bug, there was a pointer to: http://forums.sun.com/thread.jspa?threadID=306461 but this is returning a 404 (not found), so I don't know what this is referring to. There was a lot of conflicting information in the bug report and emails, so I'm not really sure exactly what the server configuration is here. Lawrence Chow wrote: > I also had to add the 3 jsse jars to the classpath BEFORE the > weblogic jars. The bug report 6998053 says this server is a BEA Weblogic 6.1 using a 1.3.1 JDK, and for SSL/TLS it appears to be using the *VERY* old unbundled version of JSSE 1.0.X that hasn't been supported in years (i.e. "the 3 jsse jars"). If that's the config, JSSE 1.0.X does not support TLS extensions and would fail but for different reasons. I went through every code path and I can't find any instances where a server would throw an illegal_parameter alert to the client. Also, the SessionID size seen in the handshake is not right size that would be sent from any of our releases. SessionID's can be 0 to 32 bytes long in TLS, and only 16 bytes in SSLv2. Every release of JSSE would return a 32 byte session id from the server, but the one returned from the first negotiation is 16 bytes. This leads me to believe the server is actually something else. Digging through the very long email, it indicated that the server might be Apache HTTPD 2, using some release of OpenSSL 0.9.8, which wouldn't surprise me given the handshaking information. For why things are failing in one case and not the other. I went through the initial ClientHellos very carefully. There's a couple concepts we need to discuss first: Hello Extensions. TLS uses TLS extensions to tell the server what ECC parameters it can/will support. SSLv2Hello: this compatibility switch converts SSLv3/TLS hellos into something that an older SSLv2 server could parse. SSLv2Hello's do not have HelloExtensions and thus whenever this operational mode is on, any Extensions are stripped from the Hello. Here's what I observed in the handshake logs on cores2.central On the first connection, SSLv2Hello is apparently on, and the ECC extensions are removed from the hello. The server never sees the extensions, so it allows the connection. On the *SECOND* negotiation, something has turned the SSLv2Hello off, and the client sends the full complement of Hello Extensions for ECC. The server is likely seeing these and keeling over. OpenSSL did not support Hello Extensions until 0.9.8f (Oct 2007), and continues to add additional support in later releases. Apparently, TLS extensions were enabled by default in 0.9.8j (Jan 2009) If this production server has an earlier version of OpenSSL than the dev machine, things will fail, likely producing that illegal_parameter error. Questions: 1) What is the exact configuration of the server? If Weblogic/JSSE, we need the java debug output. ( -Djavax.net.debug=all ) Again, I don't think this is the config. If Apache, do you have different versions of OpenSSL installed? That would explain why dev worked and production failed. 2) I'm pretty sure something is turning off SSLv2Hello on subsequent connections. Look for calls to SSLSocket.setEnabledProtocols(); Maybe once the client knows that TLS is supported on a particular server, and it enables only TLS, turning off SSLv2Hello? We're not the only ones sending Extensions, so if that's the case, they definitely need to update their server or other clients will also fail. Hope this helps. Brad
06-11-2010