FULL PRODUCT VERSION :
ADDITIONAL OS VERSION INFORMATION :
Windows, Linux
A DESCRIPTION OF THE PROBLEM :
There is a problem using PKCS11 for smartcard driver in java 8.
It seems that check regarding key size based on C_GetMechanismInfo was added and in our case PKCS11 smart card driver returns min and max key size with a value of 0 for RSA. And now because of added checkKeySize it fails, because it contains check:
        if ((minKeySize != -1) && (keySize < minKeySize)) {
            throw new InvalidKeyException(keyAlgo +
                " key must be at least " + minKeySize + " bits");
        }
        if ((maxKeySize != -1) && (keySize > maxKeySize)) {
            throw new InvalidKeyException(keyAlgo +
                " key must be at most " + maxKeySize + " bits");
        }
Error is:
Exception in thread "main" java.security.InvalidKeyException: RSA key must be at most 0 bits
	at com.sun.security.pkcs11.P11Signature.checkKeySize(P11Signature.java:366)
	at com.sun.security.pkcs11.P11Signature.engineInitSign(P11Signature.java:431)
	at java.security.Signature$Delegate.engineInitSign(Signature.java:1174)
	at java.security.Signature.initSign(Signature.java:527)
	at gem_test.Test.signDocument(Test.java:140)
	at gem_test.Test.main(Test.java:126)
Relevant mechanism is CKM_SHA1_RSA_PKCS:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
In java 7 and before there was no check regarding key size.
Our code worked from java 1.4 till 1.7 and is broken in JDK 8 and JDK 9.
REGRESSION.  Last worked in version 7u80
ADDITIONAL REGRESSION INFORMATION: 
Information for provider SunPKCS11-Personal
Library info:
  cryptokiVersion: 2.20
  manufacturerID: Nexus                           
  flags: 0
  libraryDescription: Personal NG PKCS 11             
  libraryVersion: 1.01
All slots: 1000, 0, 1, 100, 101, 102, 103, 104, 105, 106
Slots with tokens: 1000, 0, 100, 101, 102, 103, 104, 105, 106
Slot info for slot 0:
  slotDescription: Gemplus USB Key Smart Card Reader 0                             
  manufacturerID: Gemplus USB Key Smart Card Reade
  flags: CKF_TOKEN_PRESENT | CKF_REMOVABLE_DEVICE | CKF_HW_SLOT
  hardwareVersion: 255.255
  firmwareVersion: 1.00
Token info for token in slot 0:
  label: Electronic ID (PIN1)            
  manufacturerID: Technology Nexus AB             
  model: Gemalto Classic 
  serialNumber: 88889398785     
  flags: CKF_RNG | CKF_LOGIN_REQUIRED | CKF_USER_PIN_INITIALIZED
  ulMaxSessionCount: 65535
  ulSessionCount: 0
  ulMaxRwSessionCount: 65535
  ulRwSessionCount: 0
  ulMaxPinLen: 16
  ulMinPinLen: 6
  ulTotalPublicMemory: CK_UNAVAILABLE_INFORMATION
  ulFreePublicMemory: CK_UNAVAILABLE_INFORMATION
  ulTotalPrivateMemory: CK_UNAVAILABLE_INFORMATION
  ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION
  hardwareVersion: 3.00
  firmwareVersion: 3.00
  utcTime: 
Mechanism Unknown 0x0000000080694434:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism Unknown 0x0000000080694435:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism CKM_RSA_X_509:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism CKM_RIPEMD160_RSA_PKCS:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism CKM_SHA512_RSA_PKCS:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism CKM_SHA384_RSA_PKCS:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism CKM_SHA256_RSA_PKCS:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism Unknown 0x0000000080000046:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism CKM_SHA1_RSA_PKCS:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism CKM_MD5_RSA_PKCS:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism CKM_RSA_PKCS:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 0 = 
Mechanism CKM_RSA_PKCS_KEY_PAIR_GEN:
  ulMinKeySize: 512
  ulMaxKeySize: 2048
  flags: 65537 = CKF_HW | CKF_GENERATE_KEY_PAIR
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
import java.io.ByteArrayInputStream;
import java.security.KeyStore;
import java.security.PrivateKey;
import java.security.Signature;
import com.sun.security.pkcs11.SunPKCS11;
public class Test3 {
	public static void main(String[] args) throws Exception {
        char[] pin = "matej24cc".toCharArray();
        String useCertAlias = "Non Repudiation";
    	
        String pkcsConf = (
                "name = Personal\n" +
                "library = \"c:/Program Files (x86)/Personal/bin/personal.dll\"\n" +
                "showInfo = true\n" +
                "slot = 0\n"
            );
        SunPKCS11 provider = new SunPKCS11(new ByteArrayInputStream(pkcsConf.getBytes()));
        
        KeyStore keyStore = KeyStore.getInstance("PKCS11", provider);
        keyStore.load(null, pin);
        
        PrivateKey privateKey = (PrivateKey) keyStore.getKey(useCertAlias, pin);
    	
        Signature signatureAlgorithm = Signature.getInstance("SHA1withRSA", provider);
        signatureAlgorithm.initSign(privateKey);
        signatureAlgorithm.update("my sample test to be signed".getBytes("UTF-8"));
        byte[] digitalSignature = signatureAlgorithm.sign();
    }
	
}
---------- END SOURCE ----------