JDK-6877740 : Memory leak in SSLSession putValue/getValue when security manager is enabled
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 6
  • Priority: P3
  • Status: Closed
  • Resolution: Not an Issue
  • OS: linux_redhat_5.0
  • CPU: x86
  • Submitted: 2009-09-01
  • Updated: 2010-09-29
  • Resolved: 2009-09-15
Related Reports
Relates :  
JDK-4918870 - Examine session cache implementation (sun.misc.Cache)
Description
FULL PRODUCT VERSION :
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux cicada 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
Must run Java security manager to trigger issue.  Even a simple policy file with a single grant { java.security.AllPermission; }; will suffice.

A DESCRIPTION OF THE PROBLEM :
SSLSessionImpl contains a Hashtable for session-specific data.  The key for this table is the SecureKey class (contained in the same file), and when the security manager is enabled, it uses an AccessControlContext as part of this key.

When running in a servlet container (or perhaps always?) the AccessControlContext is always different, so putting a value into this session cannot subsequently be retrieved.  Programs that attempt to retrieve cached information from the session get null, recalculate, and attempt to re-persist into the cache.  When this is done frequently, it can create huge memory leaks in an application.  This is exacerbated by the 24-hour default session timeout.

Encountered this bug by deploying a CXF web service into an Apache Tomcat container.  CXF requests certain JSSE properties from the request, and Tomcat attempts to cache them in the session.  I submitted a patch to Tomcat that employs the workaround I described in the Workaround field.  Tomcat bug report: https://issues.apache.org/bugzilla/show_bug.cgi?id=47744

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Enable the Java Security Manager.  Even the simplest possible policy file will be satisfactory for reproducing this bug.
Attach a memory profiler to the JVM.
During an SSL session, retrieve the SSLSession object.  Continuously put a large object into the session using the same key.  Note that calling getValue with the same key returns null.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Memory usage is constant no matter how many calls to putValue are made.
getValue returns the object that is placed into the session.
ACTUAL -
getValue returns null; memory usage increases with every call to putValue.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
N/A.  OutOfMemory if allowed to continue for a long time.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
// this bug requires a client-server test harness to duplicate.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Use a WeakHashMap that takes an SSLSession as a key instead of using the SSL session's attribute store.
Comments
PUBLIC COMMENTS It would be big problem if the access control context is different for every sevelet request. From the view of SSLSession, there is no way to protect the session bound application layer data against accessing accross protecino domains without caching an addtional identifier, the access control context acts as the identifier here. Normally, a server contains finite number of servlets (so the code source is finite), and the number of a client's principal is also finite, there are not likely exists too much different access control context. For complex server environment, if one servlet may change its protection domain internally, such as accessing a remote jar, which will generate a new thread and call SSLSession.put/getValue(), the access control context may vary accordingly. There is no way for SSLSession to handle such cases. Applications should manage the session bound data carefully. Decreasing the session life would be help to release the memory frequently. For the test case attached in https://issues.apache.org/bugzilla/show_bug.cgi?id=47744, the access control context is the same for all https/servlet request, and the hashtab of session bound application data contains only one cached entry. No memory leak here. The described memory leak may be caused by 4918870. Note that the test case should close the connection explicit before next try, otherwise, there would a lot a idle connections. Close this bug as "not a defect".
15-09-2009