JDK-8206333 : High contention on provider.getService() with large number of TLS connections
  • Type: Bug
  • Component: security-libs
  • Sub-Component: java.security
  • Affected Version: 8,9,10,11
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • OS: linux
  • CPU: x86_64
  • Submitted: 2018-07-03
  • Updated: 2022-06-30
  • Resolved: 2018-07-19
Related Reports
Duplicate :  
JDK-7092821 - java.security.Provider.getService() is synchronized and became scalability bottleneck
Duplicate :  
JDK-8206334 - With heavy load of SSL handshakes, the application gets stuck on java.security.Provider.getService(String, String)
Description
ADDITIONAL SYSTEM INFORMATION :
Linux mailarchiva-usa-h 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
All methods in Provider.getService() are synchronized. This causes significant contention. On  systems that are performing a large number of TLS connections, Provider.getService()  can take up to 32 seconds to complete.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
This issue happens in systems with large amount of TLS connections. See: https://i.stack.imgur.com/KVM79.jpg for a demo of the same.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
We do not expect to see contention in the Provider.getService class.
ACTUAL -
You can see that monitors are blocked for a long period of time.
Java Monitor Blocked
 at java.security.Provider.getService(String, String)
 at sun.security.jca.ProviderList$ServiceList.tryGet(int)
 at sun.security.jca.ProviderList$ServiceList.access$200(ProviderList$ServiceList, int)
 at sun.security.jca.ProviderList$ServiceList$1.hasNext()
 at javax.crypto.KeyGenerator.nextSpi(KeyGeneratorSpi, boolean)
 at javax.crypto.KeyGenerator.<init>(String)
 at javax.crypto.KeyGenerator.getInstance(String)
 at sun.security.ssl.JsseJce.getKeyGenerator(String)
 at sun.security.ssl.HandshakeMessage$Finished.getFinished(HandshakeHash, int, SecretKey)
 at sun.security.ssl.HandshakeMessage$Finished.<init>(ProtocolVersion, HandshakeHash, int, SecretKey, CipherSuite)
 at sun.security.ssl.ServerHandshaker.sendChangeCipherAndFinish(boolean)
 at sun.security.ssl.ServerHandshaker.clientHello(HandshakeMessage$ClientHello)
 at sun.security.ssl.ServerHandshaker.processMessage(byte, int)
 at sun.security.ssl.Handshaker.processLoop()
 at sun.security.ssl.Handshaker.process_record(InputRecord, boolean)
 at sun.security.ssl.SSLSocketImpl.readRecord(InputRecord, boolean)
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake()
 at sun.security.ssl.SSLSocketImpl.startHandshake(boolean)
 at sun.security.ssl.SSLSocketImpl.startHandshake()
 at org.subethamail.smtp.command.StartTLSCommand.execute(String, Session)
 at org.subethamail.smtp.server.CommandHandler.handleCommand(Session, String)
 at org.subethamail.smtp.server.Session.runCommandLoop()
 at org.subethamail.smtp.server.Session.run()
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run()
 at java.lang.Thread.run()

CUSTOMER SUBMITTED WORKAROUND :
There is no workaround. This is a critical performance issue in the Java libraries. The Java Provider class needs to be refactored with modern concurrent programming principles. Several bugs have been raised in the past regarding this issue, and each time it gets ignored by Oracle.

FREQUENCY : always
Comments
From submitter: We are currently running Java 1.8.0_172. This issue was reported by someone else all the way back to 2011, so the issue has been present in Java code for at least seven years. Likely, longer. Related filed Java bugs: https://bugs.java.com/view_bug.do?bug_id=7092821 https://bugs.openjdk.java.net/browse/JDK-8172827 https://bugs.openjdk.java.net/browse/JDK-7092821 https://bugs.openjdk.java.net/browse/JDK-8080273 Some additional bugs reports on other sites: https://github.com/spring-projects/spring-security-oauth/issues/796 https://github.com/jwtk/jjwt/issues/59 https://groups.google.com/a/lists.datastax.com/forum/#!topic/java-driver-user/OVIwK1ZFd3A https://stackoverflow.com/questions/27944331/why-is-the-java-method-provider-getservicestring-type-string-algorithm-synchr https://stackoverflow.com/questions/51021107/java-security-provider-getservicestring-string-java-monitor-blocked This issue concerns the synchronization of Provider class. As you can see in the source code at http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/5ea0024ba765/src/share/classes/java/security/Provider.java, just about every method is synchronized. This results in lock contention in high performance multi-threaded applications subjected to heavy load. As one can see in the JDK 10 source code, see: http://hg.openjdk.java.net/jdk10/jdk10/jdk/file/5ea0024ba765/src/share/classes/java/security/Provider.java there is little change in level of synchronization, so its highly doubtful that there is any change. Our app will not run on JDK 10, and as of yet. We are using 1.8.0_172. I will think about how to implement a test case.
04-07-2018
To submitter: Have you started observing this issue with JDK 8u172 or it was the same with the earlier JDK versions as well ? Can you please provide a test case that may help reproduce the issue at our end. Please verify with the latest JDK versions like JDK 10.0.1 and let us know if you still experience the same issue.
04-07-2018