JDK-6340201 : RedefineClasses devours memory
  • Type: Bug
  • Component: hotspot
  • Sub-Component: jvmti
  • Affected Version: 6
  • Priority: P2
  • Status: Resolved
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2005-10-21
  • Updated: 2011-02-26
  • Resolved: 2005-11-30
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other JDK 6
5.0u8Fixed 6 b62Fixed
Related Reports
Relates :  
Description
Batch calls to JVM TI RedefineClasses leaks around 200 megabytes on the redefinition of a thousand classes.

Although RedefineClasses has a serious leak in perm gen, it appears to be unrelated and completely pales in comparison to this, which is a leak in native (not heap, not perm gen) memory.

I have very good confidence that the leak is in the classes_do line:

  RC_TRACE(100, ("+CLS_DO "));

  // Adjust constantpool caches and vtables for all classes
  // that reference methods of the evolved class.
  SystemDictionary::classes_do(adjust_cpool_cache_and_vtable);

  RC_TRACE(100, ("-CLS_DO "));

The trace lines correspond ro the critical portion of the attached logs, which look like this:

  (486, 21352) RedefineClasses: ++SINGL Memory: 8k page, physical 2097152k(585656k free)
  (486, 21352) RedefineClasses: +CLS_DO Memory: 8k page, physical 2097152k(585656k free)
  (486, 21352) RedefineClasses: -CLS_DO Memory: 8k page, physical 2097152k(585048k free)
  (486, 21352) RedefineClasses: --SINGL Memory: 8k page, physical 2097152k(585048k free)

for one loop through VM_RedefineClasses::redefine_single_class.  This sample (picked at random) shows ~600k leaked on the redefine of a single class.  That this is the leak location was cross checked by commenting out the classes_do line, the leakage reduced from 200mb to 2mb.  The numbers in parens are heap and perm gen respectively (in the loading phase the log shows leakage in perm gen but that is not happenning at all with this leak).

When the instrumentation is further drilled down, it seems to show leakage spread across the four calls to adjust_method_entries (logs for this not fine grained enough and not attached).  

The amount of leakage is exponential in the number of classes redefined and surprisingly only lightly correlated with the number of loaded classes --

For 1300 loaded classes:
  Redefined     Leaked
  Classes       Memory
     10            40K
    100         4,368K
   1000       196,020K

For 300 loaded classes:
  Redefined     Leaked
  Classes       Memory
     10            32K
    100         3,728K

The exponential nature corrresponds to the exponential performance behavior of this very same line.  Which is n-cubed by my prior analysis.

Attached are:
  redef_log      A log of the attached test run on Solaris-sparc with 1200 classes redefined
  redeftest.zip  A simple test that does the identity redefine by caching on the class load hook
  nb_redef_log   What happens when NB profiler attempts to profile itself
  hr_err_pid3440 The hs_err file when it OOME ar 2gb.

The hackery used in the logs does not deserve being committed to the record, but basically is just a call to:

   os::print_memory_info(tty);

The heap info, if the lesser problem is attacked, used:

  ResourceMark rm(Thread::current());

  // Calculate the memory usage
  size_t heap_used = 0;
  size_t non_heap_used = 0;
  
  for (int i = 0; i < MemoryService::num_memory_pools(); i++) {
    MemoryPool* pool = MemoryService::get_memory_pool(i);
    MemoryUsage u = pool->get_memory_usage();
    if (pool->is_heap()) {
      heap_used += u.used();
    } else {
      non_heap_used += u.used();
    }
  }

Have fun!

Comments
SUGGESTED FIX Add ResourceMark and HandleMark objects as needed. Please see the attached 6340201-webrev-cr0.tgz file for the proposed changes.
09-11-2005

EVALUATION You learn something new everyday. instanceKlass::itable() and instanceKlass::vtable() allocate wrapper objects every time they are called. So when adjust_cpool_cache_and_vtable() calls them a wrapper object is leaked for each call. vtable() and itable() are called at most once per call to adjust_cpool_cache_and_vtable(), but adjust_cpool_cache_and_vtable() is called alot so we have a decent sized leak.
08-11-2005

EVALUATION Robert has pointed out that there may be more than one memory leak at work here. The majority of the memory leak is due to a missing ResourceMark in: src/share/vm/prims/jvmtiRedefineClasses.cpp: VM_RedefineClasses::adjust_cpool_cache_and_vtable(): { // the previous versions' constant pool caches may need adjustment PreviousVersionWalker pvw(ik); for (PreviousVersionInfo * pv_info = pvw.next_previous_version(); pv_info != NULL; pv_info = pvw.next_previous_version()) { other_cp = pv_info->prev_constant_pool_handle(); cp_cache = other_cp->cache(); if (cp_cache != NULL) { cp_cache->adjust_method_entries(_old_methods, _new_methods); } } } // pvw is cleaned up The PreviousVersionInfo objects returned via the PreviousVersionWalker contain a GrowableArray of handles. These GrowableArrays are normally cleaned up by the nearest ResourceMark. There is a ResourceMark in the calling path of this code, but it doesn't seem to be doing the trick. I added one to the beginning of VM_RedefineClasses::doit(), but that didn't do the trick either. It occurred to me that those GrowableArrays are useless after the PreviousVersionWalker is destroyed so I looked at adding a ResourceMark field in addition to the HandleMark field that I already put in the PreviousVersionWalker. Can't do that because the GrowableArray code says when you have a GrowableArray of handles, you have to destroy the handles before the GrowableArray. I don't think there is a way to insure the order in which field destructors are called so we have to add ResourceMark objects to all the places where PreviousVersionWalker objects are created. ETA: It is probably safe to add the ResourceMark field to PreviousVersionWalker because the warning in the GrowableArray code is for the situation where the HandleMark frees the handles and the GrowableArray is still around to access those now stale handles. With PreviousVersionWalker, the intention is to destroy both fields since we are done with them and there is no other way to access to GrowableArrays. So while it should be safe to add the ResourceMark field, it doesn't work because it requires that instanceKlass.hpp depend on resourceArea.hpp and that causes an include loop. I would have to move the PreviousVersionWalker stuff to another file and I don't want to do that. ETA: Adding a ResourceMark to the top of VM_RedefineClasses::doit() didn't work. Adding one just before the loop that calls redefine_single_class() doesn't do it either. Adding one in the loop just before the call to redefine_single_class() does the trick. That doesn't make any sense, but...
08-11-2005

EVALUATION This is horrific and must be fixed in Mustang FCS and a Tiger update.
21-10-2005