Thomas Wuerthinger is working in the RedefineClasses and
RetransformClasses area so I asked him to review the fix
for the following bug:

    7122253 2/3 Instrumentation.retransformClasses() leaks class bytes

During our e-mail discussions, Thomas raised some possible issues
with my fix for 7122253 and I wanted to document those here. The
next note contains part of the e-mail thread that I think captures
the potential issue.
One other thing has occurred to me as I'm filing this bug.
Prior to the fix to 7122253, we have a memory leak. I think
we also have a problem where a RetransformClasses() call
does not cause the proper "initial class bytes" to be passed
to the ClassFileLoadHook event handler. I think this particular
bug is independent of any RedefineClasses() calls and would
occur on the second (and subsequent) RetransformClasses() calls.

Of course, we would need a test to prove this issue and to
verify that we don't have that problem after the fix for
7122253 is applied.
Thomas,

I've finished running through scenarios. Below is what I have so far.

I agree that the cached class bytes should have been "reset" when a
RedefineClasses() goes through the system, but there's a problem or
a risk that we could end up caching class bytes that have not been
validated. We don't want to do that.

Do you concur that this bad/unexpected cache interaction only
happens when RedefineClasses() and RetransformClasses() are mixed?

If that's the case, then I think we can go with my current fix for
the memory leak and file a new bug.

Please let me know what you think.

Dan



JVM/TI RetransformClasses() spec extract
----------------------------------------

> When classes are initially loaded or when they are redefined, the
> initial class file bytes can be transformed with the ClassFileLoadHook
> event. This function reruns the transformation process (whether or not
> a transformation has previously occurred). This retransformation follows
> these steps:
>
>     * starting from the initial class file bytes
>
>     * for each retransformation incapable agent which received a
>       ClassFileLoadHook event during the previous load or redefine, the
>       bytes it returned (via the new_class_data parameter) are reused
>       as the output of the transformation; note that this is equivalent
>       to reapplying the previous transformation, unaltered. except that
>       the ClassFileLoadHook event is not sent to these agents
>
>     * for each retransformation capable agent, the ClassFileLoadHook
>       event is sent, allowing a new transformation to be applied
>
>     * the transformed class file bytes are installed as the new definition
>       of the class
>
> See the ClassFileLoadHook event for more details.
>
> The initial class file bytes represent the bytes passed to
> ClassLoader.defineClass or RedefineClasses (before any transformations
> were applied), however they may not exactly match them.


Scenarios
---------

1)  Redefine first, Retransform second, Retransform...

    When Redefine happens first, class bytes are not cached on the
    instanceKlass. When the Retransform happens, it caches the class
    bytes from the previous Redefine on the instanceKlass.

    Subsequent Retransform calls will also start with the class bytes
    from the previous Redefine.

    I think this scenario works as expected.

2)  Redefine first, Retransform second, Redefine third, Redefine ...

    Same as scenario #1 until the third operation (Redefine instead of
    Retransform). The second Redefine call replaces the class bytes
    being executed, but it does not update the cached class bytes on
    the instanceKlass.

    Also note that the second (and subsequent) Redefine calls result in a
    CFLH event being posted to the Retransform capable agent. That agent
    will be given the class bytes from the Redefine call to transform.

    Subsequent Redefine operations will work as expected. Each Redefine
    will apply 'new' class bytes as transformed by the Retransform agent.

    I think this scenario works as expected.

3)  Redefine first, Retransform second, Redefine third, Retransform fourth

    Same as scenario #2 until the fourth operation (Retransform instead of
    Redefine). Again the second Redefine call replaces the class bytes
    being executed, but it does not update the cached class bytes on
    the instanceKlass. This means that second Retransform call starts
    with the class bytes from the first Redefine operation instead of
    the class bytes from the second Redefine operation.

    Also note that the second (and subsequent) Redefine calls result in a
    CFLH event being posted to the Retransform capable agent. That agent
    will be given the class bytes from the Redefine call to transform.

    Subsequent Redefine operations will work as expected; Each Redefine
    will apply 'new' class bytes as transformed by the Retransform
    agent. However, subsequent Retransform operations will always start
    with the class bytes from the first Redefine operation instead of
    the class bytes from the previous Redefine operation.

    I think this scenario shows a problem.

    When the second Redefine operation happens, the class bytes cache
    needs to be updated with the class bytes from the Redefine operation
    before they are modified by the Transform capable agent's CFLH
    handler. Note: one problem is that these class bytes haven't been
    verified yet.

4)  Retransform first, Redefine second, Retransform third, Retransform ...

    When Retransform happens first, it caches the class bytes from
    before the transform operation on the instanceKlass. The Redefine
    call replaces the class bytes being executed, but it does not update
    the cached class bytes on the instanceKlass.

    Also note that the Redefine call results in a CFLH event being posted
    to the Retransform capable agent. That agent will be given the class
    bytes from the Redefine call to transform.

    Subsequent Retransform operations will always start with the class
    bytes from before the first Retransform operation instead of the
    class bytes from the first Redefine operation.

    I think this scenario shows a problem.

    When the second operation (a Redefine) happens, the class bytes cache
    needs to be updated with the class bytes from the Redefine operation
    before they are modified by the Transform capable agent's CFLH
    handler. Note: one problem is that these class bytes haven't been
    verified yet.


On 12/21/11 2:25 PM, Thomas Wuerthinger wrote:
> Dan,
>
> Thanks for keeping me informed on that.
>
> I had a quick look over the code, and I'm not sure if you may keep the cached class file bytes in the following scenario:
> - Class C is loaded.
> - C is retransformed to C1 (thus saving cached class bytes)
> - C1 is redefined to C2 (but without any "transformation" on it, so there are no new cached class bytes)
> - C2 is redefined to C3: Here will see the transformer the class bytes from C1 (because they are the last ones cached), but it should see the class bytes from C2
> If you think that this could be problem, I can try to create a test case for it.
>
> In general, I'm wondering how big the performance gain from caching the class bytes is. I'd guess that the class file reconstitutor works quite fast.
>
> - thomas