The C_SignFinal API, according to the PKCS#11 standard, is reasonably expected to return a CKR_BUFFER_TOO_SMALL error when the allocated buffer to hold the signature does not have an enough size [1]. Note that this is the only error that would not terminate an ongoing signature operation (in case of multi-part), so it can be retried with the right size (returned from the first call by means of an in/out argument). However, the NSS Software Token implementation (checked on NSS v3.67) does not return a CKR_BUFFER_TOO_SMALL error in the aforementioned scenario when the mechanism is CKM_DSA_SHA1 -or others-, and terminates the operation [2] [3] [4] [5] [6].

For DSA signatures, the signature length is expected to be twice the size of 'q'. Thus, for a 256-bits 'q', the signature will be 64 bytes long.

If C_SignFinal is called with a 40 bytes signature buffer -or any value below the expected one-, dsa_SignDigest will detect the error and a SEC_ERROR_INVALID_ARGS error will be returned (which is then translated to a CKR_ARGUMENTS_BAD error for the API caller) [7].

What JDK-11 is doing for DSA keys in P11Signature is setting the expected signature length to be 40 bytes from the Java side [8]. On the native side, the first call to C_SignFinal is tried with that size but, for the reasons described above, the expected CKR_BUFFER_TOO_SMALL error is not seen and the 2nd call is never tried [9] [10].

JDK-16+ is not affected by this bug because, as part of the JDK-8242332 enhancement, it was fixed [11] [12]. The fix works because when the expected size is set to 0 from the Java side, the native side will set the expected size to the maximum before the call to C_SignFinal [13] [14].

Note: releases between 11 and 16 may be affected by this bug.

Even when backporting JDK-8242332 will indirectly fix this bug, JDK-8242332 is a large enhancement affecting APIs and the decision to take it to 11u (or previous releases) involves different risks, effort, priority, etc. so it should be independent in my view.

--
[1] - https://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html
[2] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L2894
[3] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L2052
[4] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L3319
[5] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L3330
[6] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/softoken/pkcs11c.c#L3348
[7] - https://github.com/nss-dev/nss/blob/8c2610c237ed23d5ce99ad640fa1d4d5167fc3e8/lib/freebl/dsa.c#L338
[8] - https://github.com/openjdk/jdk11u/blob/d1541f3012754e0bbda2524bb865c6a971fc4574/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Signature.java#L634
[9] - https://github.com/openjdk/jdk11u/blob/d1541f3012754e0bbda2524bb865c6a971fc4574/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/p11_sign.c#L252
[10] - https://github.com/openjdk/jdk11u/blob/d1541f3012754e0bbda2524bb865c6a971fc4574/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/p11_sign.c#L259
[11] - https://bugs.openjdk.java.net/browse/JDK-8242332
[12] - https://github.com/openjdk/jdk/commit/78be334c#diff-c6f6f725937a727ae58024887255fb326313a422d767c78b1bb62ed50c8c8ef8L627
[13] - https://github.com/openjdk/jdk/blob/78be334c3817a1b5840922a9bf1339a40dcc5185/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/P11Signature.java#L613
[14] - https://github.com/openjdk/jdk/blob/78be334c3817a1b5840922a9bf1339a40dcc5185/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/p11_sign.c#L241