Batch calls to JVM TI RedefineClasses leaks around 200 megabytes on the redefinition of a thousand classes.

Although RedefineClasses has a serious leak in perm gen, it appears to be unrelated and completely pales in comparison to this, which is a leak in native (not heap, not perm gen) memory.

I have very good confidence that the leak is in the classes_do line:

  RC_TRACE(100, ("+CLS_DO "));

  // Adjust constantpool caches and vtables for all classes
  // that reference methods of the evolved class.
  SystemDictionary::classes_do(adjust_cpool_cache_and_vtable);

  RC_TRACE(100, ("-CLS_DO "));

The trace lines correspond ro the critical portion of the attached logs, which look like this:

  (486, 21352) RedefineClasses: ++SINGL Memory: 8k page, physical 2097152k(585656k free)
  (486, 21352) RedefineClasses: +CLS_DO Memory: 8k page, physical 2097152k(585656k free)
  (486, 21352) RedefineClasses: -CLS_DO Memory: 8k page, physical 2097152k(585048k free)
  (486, 21352) RedefineClasses: --SINGL Memory: 8k page, physical 2097152k(585048k free)

for one loop through VM_RedefineClasses::redefine_single_class.  This sample (picked at random) shows ~600k leaked on the redefine of a single class.  That this is the leak location was cross checked by commenting out the classes_do line, the leakage reduced from 200mb to 2mb.  The numbers in parens are heap and perm gen respectively (in the loading phase the log shows leakage in perm gen but that is not happenning at all with this leak).

When the instrumentation is further drilled down, it seems to show leakage spread across the four calls to adjust_method_entries (logs for this not fine grained enough and not attached).  

The amount of leakage is exponential in the number of classes redefined and surprisingly only lightly correlated with the number of loaded classes --

For 1300 loaded classes:
  Redefined     Leaked
  Classes       Memory
     10            40K
    100         4,368K
   1000       196,020K

For 300 loaded classes:
  Redefined     Leaked
  Classes       Memory
     10            32K
    100         3,728K

The exponential nature corrresponds to the exponential performance behavior of this very same line.  Which is n-cubed by my prior analysis.

Attached are:
  redef_log      A log of the attached test run on Solaris-sparc with 1200 classes redefined
  redeftest.zip  A simple test that does the identity redefine by caching on the class load hook
  nb_redef_log   What happens when NB profiler attempts to profile itself
  hr_err_pid3440 The hs_err file when it OOME ar 2gb.

The hackery used in the logs does not deserve being committed to the record, but basically is just a call to:

   os::print_memory_info(tty);

The heap info, if the lesser problem is attacked, used:

  ResourceMark rm(Thread::current());

  // Calculate the memory usage
  size_t heap_used = 0;
  size_t non_heap_used = 0;
  
  for (int i = 0; i < MemoryService::num_memory_pools(); i++) {
    MemoryPool* pool = MemoryService::get_memory_pool(i);
    MemoryUsage u = pool->get_memory_usage();
    if (pool->is_heap()) {
      heap_used += u.used();
    } else {
      non_heap_used += u.used();
    }
  }

Have fun!