FULL PRODUCT VERSION :
java version "1.6.0_19"
Java(TM) SE Runtime Environment (build 1.6.0_19-b04)
Java HotSpot(TM) Client VM (build 16.2-b04, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows XP [Version 5.1.2600]  - x86

A DESCRIPTION OF THE PROBLEM :
Java 6u18 release notes marks bug 6753664  (jce / sunmscapi / Support SHA256 and higher in SunMSCAPI) as fixed, but it still won't work.

It seems that some changes were made on sunmscapi.jar to support SHA-256 and higher, but that sunmscapi.dll still don't support them.

Tested with 6u18 and 6u19. I've checked that I'm using the right DLL (6.0.19) and that there isn't any older sunmscapi.dll on the PATH. Tested on two different computers (both with Windows XP x86 Service Pack 3).

  Bug should be still opened or the right behavior should be implemented.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
It can be tested with this simple test code:

 Provider p = new SunMSCAPI();
 Security.addProvider(p);
 KeyStore ks = KeyStore.getInstance("WINDOWS-MY");
 ks.load(null, null);
 String alias = ks.aliases().nextElement();
 Signature s = Signature.getInstance("SHA256withRSA", p);
 s.initSign(
   ((PrivateKeyEntry)ks.getEntry(alias, new KeyStore.PasswordProtection(null))).getPrivateKey()
 );
 s.update("Hola".getBytes());
 System.err.println(s.sign());


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
A valid signature.
ACTUAL -
Exception in thread "main" java.security.SignatureException: Invalid algorithm specified.

	at sun.security.mscapi.RSASignature.signHash(Native Method)
	at sun.security.mscapi.RSASignature.engineSign(RSASignature.java:279)
	at java.security.Signature$Delegate.engineSign(Signature.java:1128)
	at java.security.Signature.sign(Signature.java:522)
	at es.atosorigin.Test.main(Test.java:30)

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Exception in thread "main" java.security.SignatureException: Invalid algorithm specified.

	at sun.security.mscapi.RSASignature.signHash(Native Method)
	at sun.security.mscapi.RSASignature.engineSign(RSASignature.java:279)
	at java.security.Signature$Delegate.engineSign(Signature.java:1128)
	at java.security.Signature.sign(Signature.java:522)
	at es.atosorigin.Test.main(Test.java:30)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
 Provider p = new SunMSCAPI();
 Security.addProvider(p);
 KeyStore ks = KeyStore.getInstance("WINDOWS-MY");
 ks.load(null, null);
 String alias = ks.aliases().nextElement();
 Signature s = Signature.getInstance("SHA256withRSA", p);
 s.initSign(
   ((PrivateKeyEntry)ks.getEntry(alias, new KeyStore.PasswordProtection(null))).getPrivateKey()
 );
 s.update("Hola".getBytes());
 System.err.println(s.sign());
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
None, you cannot use SHA512withRSA, SHA384withRSA nor SHA256withRSA with SunMSCAPI provider.
Did a bit of minor tweaking to the sample:

        KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA", "SunMSCAPI");
        kpg.initialize(1024);
        KeyPair kp = kpg.generateKeyPair();

        PrivateKey pk = kp.getPrivate();

        Signature s = Signature.getInstance("SHA256withRSA", "SunMSCAPI");
        s.initSign(pk);
        s.update("Hola".getBytes());
        System.err.println(s.sign());

I'm running on WinXP SP3.