JDK-6748284 : LDAP pooling creates multiple pools for the same credentials under load.
  • Type: Bug
  • Component: core-libs
  • Sub-Component: javax.naming
  • Affected Version: 6
  • Priority: P3
  • Status: Resolved
  • Resolution: Won't Fix
  • OS: solaris_2.5.1
  • CPU: x86
  • Submitted: 2008-09-15
  • Updated: 2017-10-27
  • Resolved: 2017-10-27
Related Reports
Relates :  
Description
FULL PRODUCT VERSION :
java version "1.6.0_04"
Java(TM) SE Runtime Environment (build 1.6.0_04-b12)
Java HotSpot(TM) Client VM (build 10.0-b19, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
1. Linux 2.6.18-53.el5PAE #1 SMP Wed Oct 10 16:48:18 EDT 2007 i686 i686 i386 GNU/Linux
2. SunOS t1003 5.10 Generic_125100-07 sun4v sparc SUNW,Sun-Fire-T1000
3. Microsoft Windows XP [Version 5.1.2600]

A DESCRIPTION OF THE PROBLEM :
LDAP pooling creates multiple pools for the same credentials beyond the limit set by "com.sun.jndi.ldap.connect.pool.maxsize".

One interested can see it monitoring tcpview or tcpdump.

If you specify
System.setProperty("com.sun.jndi.ldap.connect.pool.initsize", "2");
System.setProperty("com.sun.jndi.ldap.connect.pool.maxsize", "2");
and launch 2 threads sending LDAP requests to any LDAP server (load 200 TPS in my case) then very soon you will see there are 4 open TCP connections to this server, then 6 and so on and so forth.

My investigation revealed that LDAP pooling periodically lost current connection pool and opened a new one.

In real production system with 30 connections in pool it leads to significant performance degradation and denial of service when all 30 connections are being reestablished.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Launch any LDAP server. I used OpenLdap.

2. Specify
System.setProperty("com.sun.jndi.ldap.connect.pool.initsize", "2");
System.setProperty("com.sun.jndi.ldap.connect.pool.maxsize", "2");
and launch 2 threads sending LDAP requests to this LDAP server.
My load was 200 extended requests per second.

3. Monitor TCP connections opened by JNDI LDAP to the LDAP server.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
We expect that SUN JNDI won't establish new and new connections beyond the limit set in "com.sun.jndi.ldap.connect.pool.maxsize" leading to significant performance degradation and denial of service.
ACTUAL -
SUN JNDI establishes new and new connections beyond the limit set in "com.sun.jndi.ldap.connect.pool.maxsize" leading to significant performance degradation and denial of service.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
public static void test18361() throws Exception {
        final LdapCtxFactory fac = new LdapCtxFactory();
        final Properties env = new Properties();
        env.setProperty(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
        env.setProperty("java.naming.ldap.version", "3");
        env.setProperty("java.naming.ldap.derefAliases", "never");
        env.setProperty("com.sun.jndi.ldap.connect.timeout", "3000");
        env.setProperty("com.sun.jndi.ldap.connect.pool", "true");
        //System.setProperty("com.sun.jndi.ldap.connect.pool.debug", "all");
        System.setProperty("com.sun.jndi.ldap.connect.pool.authentication", "none simple");
        System.setProperty("com.sun.jndi.ldap.connect.pool.protocol", "plain");
        System.setProperty("com.sun.jndi.ldap.connect.pool.initsize", "2");
        System.setProperty("com.sun.jndi.ldap.connect.pool.maxsize", "2");
        System.setProperty("com.sun.jndi.ldap.connect.pool.prefsize", "0");
        System.setProperty("com.sun.jndi.ldap.connect.pool.timeout", "0");
        env.put(Context.PROVIDER_URL, "ldap://narva.jnetx.ru:389");
        env.put(Context.SECURITY_AUTHENTICATION, "simple");
        env.put(Context.SECURITY_PRINCIPAL, "cn=Manager,dc=jnetx,dc=ru");
        env.put(Context.SECURITY_CREDENTIALS, "secret");

        for (int i = 1; i < 3; ++i) {
            new Thread("Thread-" + i + "-test18361") {
                public void run() {
                    long t = System.currentTimeMillis();
                    for (int j = 0; j < Integer.MAX_VALUE; j++) {
                        try {
                            LdapContext ctx = (LdapContext)fac.getInitialContext(env);
                            ExtReq req = new ExtReq("1.3.6.1.4.1.4203.1.11.3", null); //WHOAMI OpenLdap extension
                            ExtRsp rsp = (ExtRsp)ctx.extendedOperation(req);
                            //System.out.println(rsp);
                            ctx.close();
                            Thread.sleep(5);
                            if (j > 0 && 0 == j % 100) {
                                System.out.println(Thread.currentThread().getName() + " " + j + " requests were sent, " + (System.currentTimeMillis() - t) / 1000.0 + " sec");
                            }
                        } catch (Throwable th) {
                            th.printStackTrace(System.out);
                        }
                    }
                }
            }.start();
        }

        Thread.sleep(2 * 60 * 60 * 1000);
    }

    public static class ExtReq implements ExtendedRequest {
        private final String name;
        private byte[] value;

        public ExtReq(String name, byte[] value) {
            this.name = name;
            this.value = value;
        }

        public byte[] getEncodedValue() {
            return value;
        }

        public String getID() {
            return name;
        }

        public ExtendedResponse createExtendedResponse(String id, byte[] berValue, int offset, int length) throws NamingException {
            ExtRsp resp = new ExtRsp(this, 0);
            resp.setName(id);
            if ((length > 0) && (berValue != null)) {
                byte[] value = new byte[length];
                System.arraycopy(berValue, offset, value, 0, length);
                resp.setResponse(value);
            }
            return resp;
        }

        @Override
        public String toString() {
            return new StringBuilder("ExtReq")
                    .append("{name='").append(name).append('\'')
                    .append(", value=").append(value == null ? "null" : Arrays.toString(value))
                    .append('}')
                    .toString();
        }
    }

    public static class ExtRsp implements ExtendedResponse {
        private String name;
        private byte[] response;
        final int result;
        final ExtReq request;

        public ExtRsp(ExtReq request, int result) {
            this.result = result;
            this.request = request;
        }

        public void setName(String name) {
            this.name = name;
        }

        public void setResponse(byte[] respValue) {
            response = respValue;
        }

        public String getID() {
            return name;
        }

        public byte[] getEncodedValue() {
            return response;
        }

        @Override
        public String toString() {
            return new StringBuilder("ExtRsp")
                    .append("{name='").append(name).append('\'')
                    .append(", response=").append(response == null ? "null" : Arrays.toString(response))
                    .append(", result=").append(result)
                    .append(", request=").append(request)
                    .append('}')
                    .toString();
        }
    }

    public static void main(String[] args) throws Exception {
        test18361();
    }
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
My investigation shows there is a bug in com.sun.jndi.ldap.pool.Pool class. It uses WeakHashMap for storing pools of LDAP connections and nobody holds a reference to them from the outside.

Under load GC very soon cleans those entries with LDAP pools from this map this is why the next request does not find this pool in the map and creates a new pool with new connections.

Comments
no recent activity hence closing it. Please feel free to re poen if required.
27-10-2017