United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-6845161 Bottleneck in Configuration.getConfiguration synchronized call
JDK-6845161 : Bottleneck in Configuration.getConfiguration synchronized call

Details
Type:
Bug
Submit Date:
2009-05-26
Status:
Resolved
Updated Date:
2011-02-16
Project Name:
JDK
Resolved Date:
2009-06-29
Component:
security-libs
OS:
solaris_10
Sub-Component:
javax.security
CPU:
sparc
Priority:
P1
Resolution:
Fixed
Affected Versions:
5.0u19
Fixed Versions:
5.0u19-rev (b05)

Related Reports
Backport:
Backport:
Backport:
Backport:
Backport:

Sub Tasks

Description
See related CR filed for Access Manager : CR 6844567.
Customers are experiencing diminished throughput in Access Manager / OpenSSO products on T51xx / T52xx machines (high concurrency). CPU util on those machines are capped at 10~20%.

All thread dumps, generated with kill -3, show on average 80 threads waiting on a lock to be released in :

javax.security.auth.login.Configuration.getConfiguration

The stack trace below shows the thread that currently has the lock. Most of the other worker threads are waiting for a monitor entry, as shown below :

"service-j2ee" daemon prio=3 tid=0x01e7b060 nid=0x137 waiting for monitor entry [0x32d0d000..0x32d0f7d8]
        at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:190)
        - waiting to lock <0xec228f50> (a java.lang.Class)
        at com.sun.identity.authentication.service.AMLoginContext.getModuleFromAuthConfiguration(AMLoginContext.java:1804)

I saw a bug / fix regarding the PolicyFile.implies call, but moving to update 19 doesn't seem to change anything.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6552236

It looks to me that the synchronized getConfiguration() is a major bottleneck in highly multi-threaded environment.


"service-j2ee" daemon prio=3 tid=0x0294ac88 nid=0x12f runnable [0x32f8d000..0x32f8f7d8]
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.provider.PolicyFile.addPermissions(PolicyFile.java:1370)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1325)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1291)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1227)
        at sun.security.provider.PolicyFile.implies(PolicyFile.java:1182)
        at com.sun.enterprise.security.provider.PolicyWrapper.implies(PolicyWrapper.java:135)
        at java.security.ProtectionDomain.implies(ProtectionDomain.java:195)
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:249)
        at java.security.AccessController.checkPermission(AccessController.java:427)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
        at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:192)
        - locked <0xec228f50> (a java.lang.Class)
        at com.sun.identity.authentication.service.AuthUtils.isPureJAASModulePresent(AuthUtils.java:1399)
        at com.sun.identity.authentication.service.AMLoginContext.executeLogin(AMLoginContext.java:299)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:327)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:263)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:183)
        at com.sun.identity.authentication.UI.LoginViewBean.getLoginDisplay(LoginViewBean.java:792)
        at com.sun.identity.authentication.UI.LoginViewBean.processLoginDisplay(LoginViewBean.java:1034)
        at com.sun.identity.authentication.UI.LoginViewBean.processLogin(LoginViewBean.java:734)
        at com.sun.identity.authentication.UI.LoginViewBean.forwardTo(LoginViewBean.java:444)
        at com.iplanet.jato.ApplicationServletBase.dispatchRequest(ApplicationServletBase.java:981)

                                    

Comments
EVALUATION

Sean Mullan started looking at this issue. He suggested the following fix :

One simple fix is to change Configuration.getConfiguration so that the SecurityManager check is not done inside the lock. This is not necessary and could alleviate the problem by not locking the method while all of the policy processing is done.

1) remove synchronized keyword from getConfiguration

2) add synchronized block after the SecurityManager.checkPermission call.
                                     
2009-05-27
EVALUATION

AFter using Sean's modified Configuration class for JDK 1.5.0_19, throughput was much higher (10 to 20% gain) and CPU util is now around 29% (compared to 20%).

There are no more threads waiting to on a monitor entry (waiting to lock) for Configuration.getConfiguration.
                                     
2009-05-28



Hardware and Software, Engineered to Work Together