The goal of this enhancement is to allow the initialization of the SunPKCS11 security provider with an NSS backend when there are external modules configured in FIPS mode in the NSSDB.

sun.security.pkcs11.Secmod is the class responsible for managing NSS and its configuration stored in the NSSDB database. Since the early days of OpenJDK in 2007, there is a check in-place [1] that prevents initialization when the database contains external "FIPS" modules.

The error stack trace may look like the following:

Exception in thread "main" java.lang.RuntimeException: FIPS flag set for non-internal module: p11-kit-proxy.so, p11-kit-proxy
	at jdk.crypto.cryptoki/sun.security.pkcs11.Secmod$Module.<init>(Secmod.java:418)
	at jdk.crypto.cryptoki/sun.security.pkcs11.Secmod.nssGetModuleList(Native Method)
	at jdk.crypto.cryptoki/sun.security.pkcs11.Secmod.getModules(Secmod.java:258)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:219)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$1.run(SunPKCS11.java:112)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$1.run(SunPKCS11.java:109)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
	at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.configure(SunPKCS11.java:109)
	at PKCS11Test.getSunPKCS11(PKCS11Test.java:160)
	at PKCS11Test.testNSS(PKCS11Test.java:580)
	at PKCS11Test.main(PKCS11Test.java:220)
	at PKCS11Test.main(PKCS11Test.java:196)
	at ExternalFipsModules.main(ExternalFipsModules.java:31)

Example of an NSSDB that caused the previous error:

[user@vmrheltarget ~]# modutil -dbdir /etc/pki/nssdb -list

Listing of PKCS #11 Modules
-----------------------------------------------------------
  1. NSS Internal Crypto Services
	   uri: pkcs11:library-manufacturer=Mozilla%20Foundation;library-description=NSS%20Internal%20Crypto%20Services;library-version=3.44
	 slots: 1 slot attached
	status: loaded

	 slot: NSS FIPS 140-2 User Private Key Services
	token: NSS FIPS 140-2 Certificate DB
	  uri: pkcs11:token=NSS%20FIPS%20140-2%20Certificate%20DB;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

  2. p11-kit-proxy
	library name: p11-kit-proxy.so
	   uri: pkcs11:library-manufacturer=PKCS%2311%20Kit;library-description=PKCS%2311%20Kit%20Proxy%20Module;library-version=1.1
-----------------------------------------------------------

Behavior exposure was increased by recent NSS changes. When the system-wide FIPS policy is set in Linux operating systems such as Fedora (see /proc/sys/crypto/fips_enabled), NSS sets "isFIPS" attribute in "struct SECMODModuleStr" to True [2] [3] [4] for every module installed in the NSSDB (whether external or internal). In addition, these systems automatically add an external p11-kit-proxy PKCS#11 module to the NSSDB (which may have slots opened).

In general, there are 3 pre-conditions to trigger this behavior: 1) there has to be an external module in the NSSDB, 2) the external module has to be configured in FIPS mode, and 3) the external module has at least 1 slot.

See previous discussion and more information here:
 * https://mail.openjdk.java.net/pipermail/security-dev/2019-December/021077.html
 * https://mail.openjdk.java.net/pipermail/security-dev/2020-January/021155.html

Due to external NSS dependencies and complex configurations, it's not in the scope of this enhancement to develop an automated regression test. With that said, we aim to provide manual configuration and testing steps.

--
[1] - http://hg.openjdk.java.net/jdk/jdk/file/9e54ea7d9cd9/src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/Secmod.java#l417
[2] - https://github.com/nss-dev/nss/blob/c1ff439ca931f53c318e7381636ed5889b3d66f1/lib/pk11wrap/secmodt.h#L49
[3] - https://github.com/nss-dev/nss/blob/a141cd68ece76118aebf8033c06d46a3692b55fe/lib/pk11wrap/pk11pars.c#L49
[4] - https://github.com/nss-dev/nss/blob/a141cd68ece76118aebf8033c06d46a3692b55fe/lib/pk11wrap/pk11pars.c#L819