(Issue was first found and analyzed by Ralf Schmelter at SAP. Thanks to him for his work.)

At the end of JDWP invoke handling we want to release all global references the invoke may have created. This may also  include the return value object ref, if return value was an object, and the exception ref, if an exception was thrown. 

Both return value object ref and exception ref need to be kept alive during the sending of the JDWP answer packet, because during sending references are translated into objectIDs and both references have to be not yet collected to do that.

The sending of the answer packet must happen outside the invokerLock, to minimize the chance for deadlocks.

Once sending is done, we enter the invokerLock again and clean up return value object ref and exception object ref.

Pseudocode:

{code}
invoker_completeInvokeRequest()
	enter invokerLock;
A	reset Thread's InvokeRequest structure: 
              request->pending = JNI_FALSE;
              request->started = JNI_FALSE;
              request->available = JNI_TRUE; /* For next time around */	
	retrieve exception ref and return value object ref from InvokeRequest structure
B       leave invokerLock();
C      send answer packet containing exception ref and return value object ref, both will be resolved to objectIDs.
	enter invokerLock;
D	release  exception ref and return value object ref *by accessing the thread request structure*
	leave invokerLock();
{code}

At (A) we clear the InvokeRequest structure of the thread for the next invoke request. 

At (B) we leave lock protection and from now on a new invoke request may be filled in immediately by the JDWP listener thread.

At (D) we release the references by taking them from the thread request structure. I think this is wrong, because the request structure may already have been filled by another completed invoke request, whose return values we now delete. If that request answer is sent after ours, it will encounter invalid object references.

==========

IMHO there are several ways to fix this:

1) move the "request->available = JNI_TRUE;" down after the answer packet was sent. But this opens the possibility for deadlocks, because while this (potentially longish) IO happens, no other invoke request can be served.

2) We do not need to access the thread request structure to release the object references contained in it. Just use local variables to keep the object references to release.

3) We also could split outStream_writeObjectRef() into two parts, one which gets the objectID, one which writes the objectID. Then, we could get the objectIDs for exception ref and return value object ref before (B), release them also before (B), and just send back the objectIDs in C.

I favour (2) though, because it is a less invasive change.