In order to establish TLS connections while operating in FIPS mode, the following cryptographic providers have to be enabled in the JCA (i.e. though java.security configuration file):

1. SunPKCS11 (with a back-end that operates in FIPS mode, such as NSS)
2. SUN (to handle X.509 certificates)
3. SunJSSE (for a TLS engine)

It turns out that after "8217835: Remove the experimental SunJSSE FIPS compliant mode" [1] changes, algorithms not implemented in SunPKCS11 but in other providers can be negotiated by the TLS engine. I.e.: RSASSA-PSS. When SunPKCS11 keys are sensitive (always the case in FIPS mode), they cannot be converted to be used by algorithms implemented in other providers. As a result, the connection cannot be established.

A possible workaround is to manually disable algorithms that meet the previous criteria. I.e.: adding "jdk.tls.disabledAlgorithms=RSASSA-PSS" to java.security configuration file. However, this workaround has several drawbacks: 1) the algorithm may be later implemented in SunPKCS11 and will remain banned or the list of disabled algorithms has to be manually updated, 2) new algorithms may meet the conditions for this bug in the future and the list of algorithms will have to be updated, and 3) it's inconvenient for end-users to apply this procedure to enable FIPS mode. A definite solution would be desirable.

Please also note that the TLS client must not offer unsupported ciphersuites to the server. I.e.: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) ciphersuite must not be offered if GCM mode is not supported by SunPKCS11 crypto provider. This is different than assuming that SunJCE is always enabled. See: http://hg.openjdk.java.net/jdk/jdk/file/4f2fd02922b1/src/java.base/share/classes/sun/security/ssl/SSLCipher.java#l492

Further information and discussion at https://mail.openjdk.java.net/pipermail/security-dev/2019-March/019469.html

--
[1] - https://bugs.openjdk.java.net/browse/JDK-8217835