JDK-8080273 : JCA Signature provider service loading bottlenecks
  • Type: Enhancement
  • Component: security-libs
  • Sub-Component: java.security
  • Affected Version: 7u80
  • Priority: P4
  • Status: Closed
  • Resolution: Won't Fix
  • Submitted: 2015-05-13
  • Updated: 2020-04-10
  • Resolved: 2020-04-10
Related Reports
Relates :  
JDK-8172827 - Lock contention on java.security.Provider.getService
Relates :  
JDK-7092821 - java.security.Provider.getService() is synchronized and became scalability bottleneck
Description
Submitter reporting high contention in security libraries. These are being observed during 100 concurrent threads load test for OAM Authentication. Sample thread dumps below.


Bottleneck 1 :
============

"Grizzly-worker(61)" daemon prio=10 tid=0x00007f2b4400d000 nid=0x64ef waiting 
for monitor entry [0x00007f2c466e5000] 
   java.lang.Thread.State: BLOCKED (on object monitor) 
at java.security.Provider.getService(Provider.java:680) 
- waiting to lock <0x000000074045d918> (a sun.security.provider.Sun) 
at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:436) 
at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375) 
at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485) 
at java.security.Signature.getInstance(Signature.java:223) 
at oracle.idm.gateway.grs.token.impl.TokenUtils.verify(TokenUtils.java:174) 
at oracle.idm.gateway.grs.token.impl.AToken.verify(AToken.java:583) 
at oracle.idm.gateway.grs.token.impl.TokenUtils.verifyAToken(TokenUtils.java:671) 


"Grizzly-worker(62)" daemon prio=10 tid=0x00007f2b5c012800 nid=0x64f0 
runnable [0x00007f2c465e4000] 
   java.lang.Thread.State: RUNNABLE 
at java.security.Provider.getService(Provider.java:680) 
- locked <0x000000074045d918> (a sun.security.provider.Sun) 
at sun.security.jca.ProviderList$ServiceList.tryGet(ProviderList.java:436) 
at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList.java:375) 
at sun.security.jca.ProviderList$ServiceList$1.hasNext(ProviderList.java:485) 
at java.security.Signature.getInstance(Signature.java:223) 
at oracle.idm.gateway.grs.token.impl.TokenUtils.sign(TokenUtils.java:152) 
at oracle.idm.gateway.grs.token.impl.AToken.sign(AToken.java:476) 
at oracle.idm.gateway.grs.token.impl.TokenUtils.createAToken(TokenUtils.java:639) 

==========


Bottleneck 2 :
============

"Grizzly-worker(92)" daemon prio=10 tid=0x00007f2b1c012000 nid=0x6902 waiting 
for monitor entry [0x00007f2c45cda000] 
   java.lang.Thread.State: BLOCKED (on object monitor) 
at sun.security.rsa.RSACore$BlindingParameters.getBlindingRandomPair(RSACore.java :388) 
- waiting to lock <0x0000000742057aa0> (a sun.security.rsa.RSACore$BlindingParameters) 
at sun.security.rsa.RSACore.getBlindingRandomPair(RSACore.java:429) 
at sun.security.rsa.RSACore.crtCrypt(RSACore.java:165) 
at sun.security.rsa.RSACore.rsa(RSACore.java:110) 
at sun.security.rsa.RSASignature.engineSign(RSASignature.java:177) 
at java.security.Signature$Delegate.engineSign(Signature.java:1180) 
at java.security.Signature.sign(Signature.java:553) 
at oracle.idm.gateway.grs.token.impl.TokenUtils.sign(TokenUtils.java:155) 
at oracle.idm.gateway.grs.token.impl.AToken.sign(AToken.java:476) 
at oracle.idm.gateway.grs.token.impl.TokenUtils.createAToken(TokenUtils.java:639) 


"Grizzly-worker(90)" daemon prio=10 tid=0x00007f2b7000c800 nid=0x68ff 
runnable [0x00007f2c461e0000] 
   java.lang.Thread.State: RUNNABLE 
at sun.security.rsa.RSACore$BlindingParameters.getBlindingRandomPair(RSACore.java :388) 
- locked <0x0000000742057aa0> (a sun.security.rsa.RSACore$BlindingParameters) 
at sun.security.rsa.RSACore.getBlindingRandomPair(RSACore.java:429) 
at sun.security.rsa.RSACore.crtCrypt(RSACore.java:165) 
at sun.security.rsa.RSACore.rsa(RSACore.java:110) 
at sun.security.rsa.RSASignature.engineSign(RSASignature.java:177) 
at java.security.Signature$Delegate.engineSign(Signature.java:1180) 
at java.security.Signature.sign(Signature.java:553) 
at oracle.idm.gateway.grs.token.impl.TokenUtils.sign(TokenUtils.java:155) 
at oracle.idm.gateway.grs.token.impl.AToken.sign(AToken.java:476) 
at oracle.idm.gateway.grs.token.impl.TokenUtils.createAToken(TokenUtils.java:639)
Comments
No plans to look at this in JDK 7u. JDK 8u recently got performance enhancements in this area via JDK-7092821 Probably worth upgrading given the short lifetime left in JDK 7 Updates.
10-04-2020
I obtained a JFR recording from submitter. Most contention is seen in this type of stack trace (during their stress tests) sun.security.rsa.RSACore$BlindingParameters.getBlindingRandomPair(BigInteger, BigInteger, BigInteger) 107,416 sun.security.rsa.RSACore.getBlindingRandomPair(BigInteger, BigInteger, BigInteger) 107,416 sun.security.rsa.RSACore.crtCrypt(byte[], RSAPrivateCrtKey, boolean) 107,416 sun.security.rsa.RSACore.rsa(byte[], RSAPrivateKey, boolean) 107,416 com.sun.crypto.provider.RSACipher.doFinal() 60,706 com.sun.crypto.provider.RSACipher.engineUnwrap(byte[], String, int) 60,706 javax.crypto.Cipher.unwrap(byte[], String, int) 60,706 oracle.idm.gateway.grs.token.impl.TokenUtils.decryptSymKey(String, PrivateKey, String) 60,706 Will need to see if the class locking can be re-worked. Avg blocking time was 48ms in their app.
09-07-2015
Bottleneck 2 is real, though. It seems to be blocking here: BlindingRandomPair getBlindingRandomPair( BigInteger e, BigInteger d, BigInteger n) { ... synchronized (this) { if (!u.equals(BigInteger.ZERO) && !v.equals(BigInteger.ZERO)) { brp = new BlindingRandomPair(u, v); if (u.compareTo(BigInteger.ONE) <= 0 || v.compareTo(BigInteger.ONE) <= 0) { // need to reset the random pair next time u = BigInteger.ZERO; v = BigInteger.ZERO; } else { u = u.modPow(BIG_TWO, n); v = v.modPow(BIG_TWO, n); } } // Otherwise, need to reset the random pair. I'm not sure how this code is supposed to work.
17-06-2015
Closing for now. Submitter can't provide a reproducer test. I'm attaching a simple testcase to show that synchronization in this area doesn't seem to be a major issue. (operations complete in < 300ms even with 1000 threads running through this code.)
18-05-2015