JDK-6928796 : PKCS11 issue fixes in JDK6u14->18 that breaks Symmetric encryption (when NSS PKCS11 provider used)
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.crypto:pkcs11
  • Affected Version: 6u18
  • Priority: P3
  • Status: Closed
  • Resolution: Not an Issue
  • OS: solaris_10
  • CPU: sparc
  • Submitted: 2010-02-23
  • Updated: 2011-02-16
  • Resolved: 2010-04-16
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 6
6-poolResolved
Description
Problem
========
It seems that since JDK6u13++, if one uses the SunPKCS11 module
with Symmetric key encyption, it is likely one will get into
PKCS11 error in java (CKR_BAD_ARGUMENTS).


Simple example:
===============
import javax.crypto.*;
import java.security.*;
public class SmallTest2 {
  public static void main(String args[])
  throws Exception {
      Provider nss = new sun.security.pkcs11.SunPKCS11(args[0]);
      Security.insertProviderAt(nss, 1);
      KeyGenerator kg = KeyGenerator.getInstance("DES");
      kg.init(56,new SecureRandom());
      System.out.println("SKeyGenerator provider: "+kg.getProvider());
      SecretKey skey =kg.generateKey();
      System.out.println("SecretKey: "+skey.getAlgorithm());
      Cipher sciph = Cipher.getInstance(skey.getAlgorithm());
      System.out.println("SCipher provider: "+sciph.getProvider());
      sciph.init(Cipher.ENCRYPT_MODE, skey);
      sciph.doFinal("SmallTesting".getBytes("UTF-8"));
      System.out.println("------------------");
      System.out.println();
  }
}

Take in a nss.cfg. (sample in
https://www.opends.org/wiki/page/EllipticCurveCryptography)

=============
TEST RESULTS: On Solaris
=============
On JDK5, it works too
On JDK6u12 and earlier: Run's OK
On JDK6u14 to 18: BROKEN
SKeyGenerator provider: SunPKCS11-NSS version 1.6
SecretKey: DES
SCipher provider: SunPKCS11-NSS version 1.6
Exception in thread "main" java.security.ProviderException: update() failed
        at sun.security.pkcs11.P11Cipher.implUpdate(P11Cipher.java:548)
        at sun.security.pkcs11.P11Cipher.engineUpdate(P11Cipher.java:448)
        at sun.security.pkcs11.P11Cipher.engineDoFinal(P11Cipher.java:476)
        at sun.security.pkcs11.P11Cipher.engineDoFinal(P11Cipher.java:462)
        at javax.crypto.Cipher.doFinal(DashoA13*..)
        at SmallTest2.main(SmallTest2.java:16)
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ARGUMENTS_BAD
        at sun.security.pkcs11.wrapper.PKCS11.C_EncryptUpdate(Native Method)
        at sun.security.pkcs11.P11Cipher.implUpdate(P11Cipher.java:501)
        ... 5 more
Example nss.cfg
===============
name=NSS
nssLibraryDirectory=/nss3125/lib
nssDbMode=noDb
attributes=compatibility
showInfo=true

Anyway, it seems that the working and and non-working diff
shows that the default KeyGenerator.getInstance(...)
is different. (which is also the workaround - eg : passing
SunJCE as the provider)

--- trace-ok.txt        Wed Mar  3 10:34:37 2010
+++ trace-fail.txt      Wed Mar  3 10:32:23 2010
@@ -500,8 +500,6 @@
   ulMinKeySize: 16
   ulMaxKeySize: 32
   flags: 393984 = CKF_ENCRYPT | CKF_DECRYPT | CKF_WRAP | CKF_UNWRAP
-SKeyGenerator provider: SunJCE version 1.6
+SKeyGenerator provider: SunPKCS11-NSS version 1.6
 SecretKey: DES
-SCipher provider: SunJCE version 1.6
-------------------
-
+SCipher provider: SunPKCS11-NSS version 1.6
=====================
Further information.
=====================
- Without using NSS as the Softoken PKCS11, things work
  (and in JDK6u18, SunPKCS11-Solaris also works)

- Now as a WORKAROUND for NSS, it either means
  that the PKCS11 module does not work with DES
  It seems that if we add 

disabledMechanisms = {
   CKM_DES_ECB
   CKM_DES_CBC
}

it works. (Specifically CKM_DES_ECB actually is needed
and the testcase will work).

Further changing the code to be 8-byte aligned
the testcase work. That means that the 
NSS does not support any algorithm for non-8-byte aligned
(or PADDING).

Comments
WORK AROUND 1. if you must use NSS, disable its problematic mechanisms, such as CKM_DES_ECB, CKM_DES3_ECB, CKM_AES_ECB. 2. otherwise, use a different PKCS11 library, such as the Solaris one. 3. or, if somehow 1 and 2 are not possible alternatives for you, you can either disable the SunPKCS11 provider or move it to the last of the provider list. In this case, you'll end up using the pure java based crypto providers, such as SunJCE or SunRsaSign.
15-04-2010

EVALUATION Well, judging from the description, this is an NSS bug. There is one enhancement in SunPKCS11 provider for supporting block ciphers with ECB mode and PKCS5Padding. DES is one of the block cipher algorithms. In older releases, the default mode for DES Cipher is ECB and SunPKCS11 provider (regardless NSS or Solaris) will not be used since SunPKCS11 provider does not delegate down to the native PKCS11 NSS/Solaris impl. It seems that this enhancement exposed an NSS bug which leads to the unexpected CKR_ARGUMENTS_BAD error. I think the problem lies with NSS and should be fixed there.
15-03-2010