JDK-6931964 : P11Mac session cleared when invoking C_SignFinal
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.crypto:pkcs11
  • Affected Version: 6u16
  • Priority: P2
  • Status: Closed
  • Resolution: Cannot Reproduce
  • OS: solaris
  • CPU: x86
  • Submitted: 2010-03-04
  • Updated: 2011-02-16
  • Resolved: 2010-10-26
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 6
6-poolResolved
Related Reports
Relates :  
JDK-6906488 - one pkcs11 operation is done by multiple different threads which causes trouble
Description
Java application crashes in Pkcs11 operations with stack trace that look like the following.

hs_err_pid731.log
C  [pkcs11_softtoken.so.1+0x1f8b5]  C_DecryptFinal+0x3125
C  [pkcs11_softtoken.so.1+0x1e2e4]  C_DecryptFinal+0x1b54
C  [pkcs11_softtoken.so.1+0xefbb]  C_SignFinal+0x8b
C  [libpkcs11.so.1+0x16254]  C_SignFinal+0x34
C  [libj2pkcs11.so+0xc704]  Java_sun_security_pkcs11_wrapper_PKCS11_C_1SignFinal
+0x64
j  sun.security.pkcs11.wrapper.PKCS11.C_SignFinal(JI)[B+0
j  sun.security.pkcs11.P11Mac.engineDoFinal()[B+19
j  javax.crypto.Mac.doFinal()[B+25
j  com.sun.net.ssl.internal.ssl.MAC.compute(BLjava/nio/ByteBuffer;[BII)[B+98
j  com.sun.net.ssl.internal.ssl.MAC.compute(B[BII)[B+7


hs_err_pid26105.log
C  [pkcs11_softtoken.so.1+0x1f455]  C_DecryptFinal+0x3125
C  [pkcs11_softtoken.so.1+0x1de84]  C_DecryptFinal+0x1b54
C  [pkcs11_softtoken.so.1+0xed3b]  C_SignFinal+0x8b
C  [libpkcs11.so.1+0x15f24]  C_SignFinal+0x34
C  [libj2pkcs11.so+0xc704]  Java_sun_security_pkcs11_wrapper_PKCS11_C_1SignFinal+0x64
j  sun.security.pkcs11.wrapper.PKCS11.C_SignFinal(JI)[B+0
j  sun.security.pkcs11.P11Mac.cancelOperation()V+54
j  sun.security.pkcs11.P11Mac.finalize()V+18
v  ~StubRoutines::call_stub

hs_err_pid14868.log

C  [pkcs11_softtoken.so.1+0x3645e]  C_DecryptFinal+0x1a12e
C  [pkcs11_softtoken.so.1+0x1d5b5]  C_DecryptFinal+0x1285
C  [pkcs11_softtoken.so.1+0x1d65b]  C_DecryptFinal+0x132b
C  [pkcs11_softtoken.so.1+0x1c24c]  C_DecryptUpdate+0xac
C  [libpkcs11.so.1+0x157b4]  C_DecryptUpdate+0x34
C  [libj2pkcs11.so+0x7fd0]  Java_sun_security_pkcs11_wrapper_PKCS11_C_1DecryptUpdate+0x120
j  sun.security.pkcs11.wrapper.PKCS11.C_DecryptUpdate(JJ[BIIJ[BII)I+0
j  sun.security.pkcs11.P11Cipher.implUpdate([BII[BII)I+247
j  sun.security.pkcs11.P11Cipher.engineUpdate([BII[BI)I+18
j  javax.crypto.Cipher.update([BII[BI)I+60
j  com.sun.net.ssl.internal.ssl.CipherBox.decrypt([BII)I+18
j  com.sun.net.ssl.internal.ssl.InputRecord.decrypt(Lcom/sun/net/ssl/internal/ssl/CipherBox;)V+16
j  com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Lcom/sun/net/ssl/internal/ssl/InputRecord;Z)V+218
j  com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Lcom/sun/net/ssl/internal/ssl/InputRecord;)V+15
j  com.sun.net.ssl.internal.ssl.AppInputStream.read([BII)I+30
j  java.io.BufferedInputStream.fill()V+175
j  java.io.BufferedInputStream.skip(J)J+47
j  sun.net.www.MeteredStream.skip(J)J+67
j  sun.net.www.http.KeepAliveStreamCleaner.run()V+109
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub

hs_err_pid25605
C  [pkcs11_softtoken.so.1+0x3645e]  C_DecryptFinal+0x1a12e
C  [pkcs11_softtoken.so.1+0x1ca75]  C_DecryptFinal+0x745
C  [pkcs11_softtoken.so.1+0x1cb1b]  C_DecryptFinal+0x7eb
C  [pkcs11_softtoken.so.1+0x1bb1c]  C_EncryptUpdate+0xac
C  [libpkcs11.so.1+0x15494]  C_EncryptUpdate+0x34
C  [libj2pkcs11.so+0x7a80]  Java_sun_security_pkcs11_wrapper_PKCS11_C_1EncryptUpdate+0x120
j  sun.security.pkcs11.wrapper.PKCS11.C_EncryptUpdate(JJ[BIIJ[BII)I+0
j  sun.security.pkcs11.P11Cipher.implUpdate([BII[BII)I+57
j  sun.security.pkcs11.P11Cipher.engineUpdate([BII[BI)I+18
j  javax.crypto.Cipher.update([BII[BI)I+60
j  com.sun.net.ssl.internal.ssl.CipherBox.encrypt([BII)I+107
j  com.sun.net.ssl.internal.ssl.OutputRecord.encrypt(Lcom/sun/net/ssl/internal/ssl/CipherBox;)V+16
j  com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(Lcom/sun/net/ssl/internal/ssl/OutputRecord;)V+13
j  com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(Lcom/sun/net/ssl/internal/ssl/OutputRecord;)V+294
j  com.sun.net.ssl.internal.ssl.SSLSocketImpl.sendAlert(BB)V+222
j  com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(BLjava/lang/String;Ljava/lang/Throwable;)V+82
j  com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(BLjava/lang/Throwable;)V+4
j  com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(Ljava/lang/Exception;Z)V+112
j  com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(Ljava/lang/Exception;)V+3
j  com.sun.net.ssl.internal.ssl.AppInputStream.read([BII)I+82
j  java.io.BufferedInputStream.fill()V+175
j  java.io.BufferedInputStream.skip(J)J+47
j  sun.net.www.MeteredStream.skip(J)J+67
j  sun.net.www.http.KeepAliveStreamCleaner.run()V+109
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub

Log files are attached.
Comments
EVALUATION Well, if the crash thread is indeed finalizer thread, it's possible that the crash may also be due to the p11 key objects being finalized before the p11 mac objects. Fixes for 6918573 removed the usage of finalizer() method for almost all of the PKCS11 classes (except P11KeyStore whose finalizer() method does not depend on any other pkcs11 classes), so perhaps it would take care of this problem also? Worthwhile to ask customer to try 6u21 to see if the problem still persists.
16-09-2010