JDK-6578658 : Request for raw RSA (NONEwithRSA) Signature support in SunMSCAPI
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.crypto
  • Affected Version: 6,6u20,6u21
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: generic,windows_xp
  • CPU: generic,x86
  • Submitted: 2007-07-10
  • Updated: 2016-02-17
  • Resolved: 2011-06-22
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availabitlity Release.

To download the current JDK release, click here.
JDK 6 JDK 7
6u27Fixed 7 b142Fixed
Related Reports
Duplicate :  
Relates :  
Relates :  
Relates :  
Description
A DESCRIPTION OF THE REQUEST :
Requesting support for doing raw RSA signing with SunMSCAPI - at present only SHA-1, MD5 and MD2 signing is possible.

This would include the SunMSCAPI provider having the following property or similar:

Signature.NONEwithRSA=sun.security.mscapi.RSASignature$NONE

Alternatively, the SunMSCAPI cipher class could be changed to support encrypting with private keys, although i suspect that the underlying MSCAPI doesn't allow that.

JUSTIFICATION :
NONEwithRSA Signing is required by JSSE for client authentication enabled SSL. At the moment because SunMSCAPI doesn't include a NONEwithRSA property, the default NONEwithRSA signature class is being used. This class merely wraps the cipher class - and the MSCAPI cipher class doesn't support encrypting with private keys. A "Bad Key" exception results.

See: http://forum.java.sun.com/thread.jspa?threadID=5188515&tstart=0

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
After setting a client application to use the Windows key and trust stores in the normal manner, and given that the server and client trust each other's certificates, using JSSE to connect to a server using SSL with client authentication should work.
ACTUAL -
The following error is thrown during the client verification stage, on the client end:

Exception in thread "main" javax.net.ssl.SSLException: java.security.ProviderException: java.security.KeyException: Bad Key.
	at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1520)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1487)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1470)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1396)
	at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:64)
	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
	at SunMSCAPITest.main(SunMSCAPITest.java:37)
Caused by: java.security.ProviderException: java.security.KeyException: Bad Key.

	at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:242)
	at sun.security.mscapi.RSACipher.engineDoFinal(RSACipher.java:266)
	at javax.crypto.Cipher.doFinal(DashoA13*..)
	at java.security.Signature$CipherAdapter.engineSign(Signature.java:1225)
	at java.security.Signature$Delegate.engineSign(Signature.java:1128)
	at java.security.Signature.sign(Signature.java:522)
	at com.sun.net.ssl.internal.ssl.RSASignature.engineSign(RSASignature.java:149)
	at java.security.Signature$Delegate.engineSign(Signature.java:1128)
	at java.security.Signature.sign(Signature.java:522)
	at com.sun.net.ssl.internal.ssl.HandshakeMessage$CertificateVerify.<init>(HandshakeMessage.java:1216)
	at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:715)
	at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:197)
	at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:511)
	at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:449)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:817)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1029)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:621)
	at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
	... 6 more
Caused by: java.security.KeyException: Bad Key.

	at sun.security.mscapi.RSACipher.encryptDecrypt(Native Method)
	at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:222)
	... 23 more

---------- BEGIN SOURCE ----------
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.net.Socket;

import javax.net.ssl.SSLSocketFactory;

public class SunMSCAPITest {
	
	// Website to access
	public static final String TARGET_HTTPS_SERVER = "localhost";
	public static final int TARGET_HTTPS_PORT = 8443;

	public static void main(String[] args) throws Exception {
		
		System.setProperty("javax.net.ssl.keyStoreProvider","SunMSCAPI");
		System.setProperty("javax.net.ssl.keyStoreType","Windows-MY");
		System.setProperty("javax.net.ssl.trustStoreProvider","SunMSCAPI");
		System.setProperty("javax.net.ssl.trustStoreType","Windows-ROOT");

		Socket socket = SSLSocketFactory.getDefault().createSocket(TARGET_HTTPS_SERVER,
				TARGET_HTTPS_PORT);
		
		try {
			Writer out = new OutputStreamWriter(socket.getOutputStream(),
					"ISO-8859-1");
			out.write("GET / HTTP/1.1\r\n");
			out.write("Host: " + TARGET_HTTPS_SERVER + ":" + TARGET_HTTPS_PORT
					+ "\r\n");
			out.write("Agent: SSL-TEST\r\n");
			out.write("\r\n");
			out.flush();
			BufferedReader in = new BufferedReader(new InputStreamReader(socket
					.getInputStream(), "ISO-8859-1"));
			String line = null;
			while ((line = in.readLine()) != null) {
				System.out.println(line);
			}
		} finally {
			socket.close();
		}
	}
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
At the moment this can be worked around by setting both the client and server applications to use one of the other signature types instead of NONEwithRSA:

i.e.
Provider p = Security.getProvider("SunMSCAPI");
p.setProperty("Signature.NONEwithRSA","sun.security.mscapi.RSASignature$SHA1");

This isn't always possible to do on the server end though, and breaks SSL compatibility.

Comments
EVALUATION I offered to help Vinnie with this, but haven't been able to figure this out. My original thought was that Windows 7 might have been enhanced so that Ciphers can take either public or private keys for encryption, but that is not the case. I was able to get certain PrivateKeys to initialize a Cipher object on XP and Windows 7. Here is a small code snippet that shows a simplified test case. import java.security.*; import java.util.Enumeration; import javax.crypto.Cipher; public class Main { public static void main(String[] args) throws Exception { Provider[] p = Security.getProviders(); KeyStore ks = KeyStore.getInstance("Windows-MY", "SunMSCAPI"); ks.load(null, null); Enumeration e = ks.aliases(); PrivateKey privateKey = null; PublicKey publicKey = null; while (e.hasMoreElements()) { String alias = (String) e.nextElement(); if (ks.isKeyEntry(alias)) { System.out.println("got it: " + alias); privateKey = (PrivateKey) ks.getKey(alias, null); publicKey = (PublicKey) ks.getCertificate(alias).getPublicKey(); } } System.out.println(privateKey); Signature sig = Signature.getInstance("NONEwithRSA", "SunMSCAPI"); sig.initSign(privateKey); sig.update((byte)0x15); byte [] ba = sig.sign(); } } For creating keypairs/certs, I used one of three methods: 1) The MS SDK makecert utility. (FAIL) % makecert c:/tmp/cert.cer -a sha1 \ -n "CN=Sun JSSE TestService, O=Java, C=US" -ss My 2) keytool (PASS) [wetmore@dualcongas] 338 >keytool -genkey -keyalg rsa -keystore foo.p12 -storety pe pkcs12 Enter keystore password: Re-enter new password: What is your first and last name? [Unknown]: Sun JSSE TestService What is the name of your organizational unit? [Unknown]: Java What is the name of your organization? [Unknown]: Sun What is the name of your City or Locality? [Unknown]: SCA What is the name of your State or Province? [Unknown]: CA What is the two-letter country code for this unit? [Unknown]: US Is CN=Brad, OU=Java, O=Sun, L=SCA, ST=CA, C=US correct? [no]: yes 3) The interoperability test suite test certs that were created for TLS 1.2 during JDK 7 development. (PASS) % cd <JDK7>/jdk/test/closed/sun/security/ssl/sanity/compatibility/helpers/certs % sh ./generate.sh I thought maybe it maybe had something to do with keyusages, but 2) doesn't produce keyusages, so I'm not seeing something. Only 1) failed to run, it failed with: [wetmore@dualcongas] 317 >java Main got it: Sun JSSE Test Service RSAPrivateKey [size=1024 bits, type=Signature, container=JoeSoft] Exception in thread "main" java.security.ProviderException: java.security.KeyExc eption: Bad Key. at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:242) at sun.security.mscapi.RSACipher.engineDoFinal(RSACipher.java:266) at javax.crypto.Cipher.doFinal(DashoA13*..) at java.security.Signature$CipherAdapter.engineSign(Signature.java:1225) at java.security.Signature$Delegate.engineSign(Signature.java:1128) at java.security.Signature.sign(Signature.java:522) at Main.main(Main.java:37) Caused by: java.security.KeyException: Bad Key. at sun.security.mscapi.RSACipher.encryptDecrypt(Native Method) at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:222) ... 6 more To do cert management while testing, it is best to use certmgr rather than using the cert manager in IE. If you use IE, go into the Internet Options, then content, certificates, personal certificates. Originally, I was thinking that Ciphers wouldn't allow initialization with PrivateKeys, so we might want to consider rewriting this using CryptSignMessage, if there is an option for NoPadding. That is, we simply have a regular Cipher *AND* Signature. But at this point, I had no problems using either 2)/3), so I'm not sure where the problem really lies. Also, when looking at this, please note the second webbug, I think the problems described are different, and should be investigated further. This seems to be true, I was getting different values on each run. public static void main(String[] args) throws Exception { Provider[] p = Security.getProviders(); KeyStore ks = KeyStore.getInstance("Windows-MY", "SunMSCAPI"); ks.load(null, null); Enumeration e = ks.aliases(); PrivateKey privateKey = null; PublicKey publicKey = null; while (e.hasMoreElements()) { String alias = (String) e.nextElement(); if (ks.isKeyEntry(alias)) { System.out.println("got it: " + alias); privateKey = (PrivateKey) ks.getKey(alias, null); publicKey = (PublicKey) ks.getCertificate(alias).getPublicKey(); } } System.out.println(privateKey); Signature sig1 = Signature.getInstance("NONEwithRSA", "SunMSCAPI"); sig1.initSign(privateKey); sig1.update((byte)0x15); byte [] ba = sig1.sign(); System.out.println("Output Length: " + ba.length*8); for (byte b : ba) { System.out.print(b + " "); } System.out.println(); Signature sig2 = Signature.getInstance("NONEwithRSA", "SunMSCAPI"); sig2.initSign(privateKey); sig2.update((byte)0x15); byte [] ba2 = sig2.sign(); System.out.println("Output Length: " + ba2.length*8); for (byte b : ba2) { System.out.print(b + " "); } System.out.println(); // using the first available sig1 = Signature.getInstance("NONEwithRSA"); sig1.initVerify(publicKey); sig1.update((byte)0x15); if (sig1.verify(ba)) { System.out.println("Verify PASSED"); } else { System.out.println("Verify FAILED"); } sig1 = Signature.getInstance("NONEwithRSA"); sig1.initVerify(publicKey); sig1.update((byte)0x15); if (sig1.verify(ba2)) { System.out.println("Verify PASSED"); } else { System.out.println("Verify FAILED"); } } } which outputs: [wetmore@dualcongas] 330 >java Main got it: CN=Brad,OU=Java,O=Sun,L=SCA,ST=CA,C=US RSAPrivateKey [size=1024 bits, type=Exchange, container={A18137D3-228C-42C9-A81C -AC29F1AC4CA7}] Output Length: 1024 59 33 -5 -115 103 -97 -123 -41 38 -93 53 -52 20 39 -51 -114 97 39 53 -1 32 48 32 -20 28 2 89 113 123 -84 -31 -68 -24 -107 -93 66 -76 -80 -47 -81 -91 79 115 -35 110 -51 -44 94 -112 114 -126 -49 -127 38 -61 -48 -89 -28 57 -74 -117 95 49 44 -4 2 -84 27 94 -90 -33 23 -7 -35 -63 -35 -15 -59 66 16 -15 12 105 -106 -86 17 122 - 4 121 -87 40 -63 -121 119 4 -3 17 112 50 -120 72 -35 -12 10 61 -34 84 111 3 -59 50 103 88 118 -102 -37 -24 65 3 -6 -3 -31 -80 -74 119 -59 -35 -119 -97 Output Length: 1024 102 62 11 -53 -78 -31 -41 -114 -127 70 -78 17 84 71 16 -32 125 27 -75 -94 -50 -2 9 -51 -98 106 39 -92 113 -78 -99 -107 -113 12 -39 98 -19 -68 -91 -6 84 99 104 -1 23 -77 -123 -52 -45 83 -6 118 21 -56 58 92 -80 27 -62 12 -12 34 -28 -24 -96 -38 73 -118 -54 35 -20 -64 -76 -26 -10 -96 77 -72 44 -74 -89 69 -91 13 -122 -89 -22 -64 72 -23 28 -11 84 -62 -90 -85 -91 -80 60 77 -55 114 -34 -10 105 -53 -66 -122 -82 -66 12 -96 63 -65 -1 -49 89 -45 -48 -93 -108 94 -120 103 22 53 -25 69 102 -2 3 Verify FAILED Verify FAILED Note the signatures are indeed different. At this point, I need to move on.
2011-02-15