This is logged by Oracle Weblogic team This is tracked in BugDB BUG 9502049 - STRESS BPM - MEMORY LEAK DETECTED. BUG 9527439 - STRESS MEMORY LEAK IN SERVERNOTIFFORWARDER repro.jar repro test case attached The WL developer who reported this issue had been dialoging with ###@###.### and the analysis from the above BUGDB bugs follows ***************************************************** Email to Eamonn --------------- Eamonn, I believe we are running into a memory leak in a long running JMX connection. I am wondering if it is a known issue. The details are as follows - WLS has an admin server and managed servers. We have implemented a federated model where the admin server delegates requests to the managed servers. The admin server has a long running JMX connection to each managed server. - a user adds a notification listener on the adminserver side of the connection (client side) - ArrayNotificationBuffer registers a listener for every object in the MBeanServer on the managed server for (ObjectName name : names) addBufferListener(name); - ArrayNotificationBuffer registers a listener to be notified about new MBeans added to the managed server addNotificationListener(MBeanServerDelegate.DELEGATE_NAME, creationListener, creationFilter, null); - ServerNotifForwarder keeps a listenerMap of notification listeners (server side) listenerMap.put(nn, set); - it does not appear that these listeners entries are removed unless the connection is closed. The ArrayNotificationBuffer does not remove them, it does not appear that they are removed if a MBean is unregistered, and the code prevents them from being removed if the MBean is not registered if (name != null && !name.isPattern()) { if (!mbeanServer.isRegistered(name)) { throw new InstanceNotFoundException("The MBean " + name + " is not registered."); } } So it appears if you have a number of short lived MBeans, then the listenerMap will continue to grow until the connection is closed. My questions are: - do you agree this is an issue with the JMX implementation? Is this a known issue? - is there anything we can do to workaround the issue? [Somehow cycling the connections is a fairly big change to make and we are at the very end of our release cycle.] - is there a mechanism for patching the JMX implementation? If you are not the right person, please let me know who to contact. Thanks, Peter *** PBOWER 03/26/10 08:47 am *** Response from ��amonn Hi Peter, Yes, I am still the right person to contact with technical questions about JMX. The former JMX team is no longer working on it full time but we are still available to help with urgent problems. I believe the behaviour you describe is deliberate. As I recall, our reasoning was that clients do not need to have an actual network connection open to be valid. For example, the connector could be using an datagram protocol or the like. Such a client should nevertheless receive notifications, and since it cannot distinguish between, on the one hand, an MBean that was unregistered and then re-registered between two of its transient connections, and on the other, an MBean that remained there the whole time, it would not know in the former case that it needed to re-add its listeners to the new MBean. The downside is the behaviour you observe when many MBeans come and go, and the client (typically a federating client) adds a listener automatically to each of them. The code you cite, that tests whether the MBean is registered before removing the listenerMap entry, does not seem correct to me. I think it should throw InstanceNotFoundException only if there is no entry in the listenerMap. It would choose between InstanceNotFoundException and ListenerNotFoundException based on whether the MBean exists. That would mean that you could remove a listener "ghost" from an MBean that has already been unregistered. It is a slight deformation of JMX semantics but does not seem unreasonable. Unfortunately, since the JMX API is part of the JDK, the only way to patch it is as part of a JDK update release. We could set the wheels in motion for this if you think it would help, but it would be months before you would see the patch, and, even then, only customers who had updated to the very latest JDK 6 would be able to take advantage of it. If the problem is very critical, then the best workaround I can think of would be to abuse reflection to stick your hand into the motor and tweak things while it's running. :-/ I'm thinking you could have a thread that accesses ServerNotifForwarder and thence listenerMap directly (via Field.setAccessible), and that periodically purges listenerMap of entries for MBeans that no longer exist. It's very nasty, and it requires your code to be privileged, but I think it would work. (I would suggest failing safe, though, so if you don't find ServerNotifForwarder or listenerMap on whatever JRE you are running on, you just ignore the problem.) Alternatively, as you suggest, you could cycle your connections periodically. Both delegation and notifications are much improved in JMX 2.0, and you would have a much better solution there, but unfortunately that has been deferred from JDK 7 so you won't be able to use it any time soon. Regards, @ ��amonn McManus �� JMX Spec Lead �� http://weblogs.java.net/blog/emcmanus ***************************************************
|