JDK-6845161 : Bottleneck in Configuration.getConfiguration synchronized call
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.security
  • Affected Version: 5.0u19
  • Priority: P1
  • Status: Resolved
  • Resolution: Fixed
  • OS: solaris_10
  • CPU: sparc
  • Submitted: 2009-05-26
  • Updated: 2011-02-16
  • Resolved: 2009-06-29
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other Other Other JDK 6 JDK 7 Other
5.0u19-rev,OpenJDK6Fixed 5.0u20-revFixed 5.0u21Fixed 6u18Fixed 7Fixed OpenJDK6Fixed
Description
See related CR filed for Access Manager : CR 6844567.
Customers are experiencing diminished throughput in Access Manager / OpenSSO products on T51xx / T52xx machines (high concurrency). CPU util on those machines are capped at 10~20%.

All thread dumps, generated with kill -3, show on average 80 threads waiting on a lock to be released in :

javax.security.auth.login.Configuration.getConfiguration

The stack trace below shows the thread that currently has the lock. Most of the other worker threads are waiting for a monitor entry, as shown below :

"service-j2ee" daemon prio=3 tid=0x01e7b060 nid=0x137 waiting for monitor entry [0x32d0d000..0x32d0f7d8]
        at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:190)
        - waiting to lock <0xec228f50> (a java.lang.Class)
        at com.sun.identity.authentication.service.AMLoginContext.getModuleFromAuthConfiguration(AMLoginContext.java:1804)

I saw a bug / fix regarding the PolicyFile.implies call, but moving to update 19 doesn't seem to change anything.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6552236

It looks to me that the synchronized getConfiguration() is a major bottleneck in highly multi-threaded environment.


"service-j2ee" daemon prio=3 tid=0x0294ac88 nid=0x12f runnable [0x32f8d000..0x32f8f7d8]
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.provider.PolicyFile.addPermissions(PolicyFile.java:1370)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1325)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1291)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1227)
        at sun.security.provider.PolicyFile.implies(PolicyFile.java:1182)
        at com.sun.enterprise.security.provider.PolicyWrapper.implies(PolicyWrapper.java:135)
        at java.security.ProtectionDomain.implies(ProtectionDomain.java:195)
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:249)
        at java.security.AccessController.checkPermission(AccessController.java:427)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
        at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:192)
        - locked <0xec228f50> (a java.lang.Class)
        at com.sun.identity.authentication.service.AuthUtils.isPureJAASModulePresent(AuthUtils.java:1399)
        at com.sun.identity.authentication.service.AMLoginContext.executeLogin(AMLoginContext.java:299)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:327)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:263)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:183)
        at com.sun.identity.authentication.UI.LoginViewBean.getLoginDisplay(LoginViewBean.java:792)
        at com.sun.identity.authentication.UI.LoginViewBean.processLoginDisplay(LoginViewBean.java:1034)
        at com.sun.identity.authentication.UI.LoginViewBean.processLogin(LoginViewBean.java:734)
        at com.sun.identity.authentication.UI.LoginViewBean.forwardTo(LoginViewBean.java:444)
        at com.iplanet.jato.ApplicationServletBase.dispatchRequest(ApplicationServletBase.java:981)
Comments
EVALUATION AFter using Sean's modified Configuration class for JDK 1.5.0_19, throughput was much higher (10 to 20% gain) and CPU util is now around 29% (compared to 20%). There are no more threads waiting to on a monitor entry (waiting to lock) for Configuration.getConfiguration.
28-05-2009
EVALUATION Sean Mullan started looking at this issue. He suggested the following fix : One simple fix is to change Configuration.getConfiguration so that the SecurityManager check is not done inside the lock. This is not necessary and could alleviate the problem by not locking the method while all of the policy processing is done. 1) remove synchronized keyword from getConfiguration 2) add synchronized block after the SecurityManager.checkPermission call.
27-05-2009