The new out-of-process Java Plug-In is capable of much more robust applet shutdown than the current in-process one. The calling of Applet.stop() and destroy(), disposal of the AppContext, termination of the applet's ThreadGroup, etc. are all essentially cooperative operations. Rogue applets can subvert these mechanisms and cause resource leaks such as thread leaks, etc. Because the new plugin has the option of terminating a given attached JVM instance and starting a new one, it can do much better.

We need to determine when applet termination was not completely successful and react in some way. The notion of "tainting" an attached JVM instance is needed. Once a JVM is tainted, no new applets should be executed in it. Once all applets running in that JVM instance have exited, it should shut itself down.

Roughly the following changes are needed to make this happen.

1. In Applet2Manager.java, stop(), we need more robust detection mechanisms if the shutdown of a given applet was unsuccessful. stop() should be changed to take as optional argument an instance of some sort of Listener (a new Applet2StopListener?) which will receive a method call (stopFailed?) if something goes wrong. The following items detail at least some of the failure modes that should be detected.
2. If the invokeLater() call disposing the parent window (typically the EmbeddedFrame) sees an exception during the call to parentWindow.dispose(), that indicates an incidence of 6592751 and stopFailed() should be called.
3. Before calling AppContext.dispose() in the newly-created background thread, we should replicate some code from it that tries to run Window.getOwnerlessWindows() or Frame.getFrames() (depending on the JDK version we're running on) on the EDT of the applet. This call needs to have the possibility of timing out in case the EDT is locked up. After calling AppContext.dispose(), we should check each Window and see whether isDisplayable() returns true. If it does, this indicates an incidence of 6592751 and stopFailed() should be called.
4. We should capture the ThreadGroup of the applet before calling AppContext.dispose(). After calling it, we should check to see if there are any threads left alive in it or any sub-ThreadGroups still present. If there are, stopFailed() should be called.
5. PluginMain.java should provide an implementation of Applet2StopListener which writes a new MarkTaintedMessage to the pipe.
6. JVMInstance.java, inner class WorkerThread, needs to process MarkTaintedMessages and mark the current JVMInstance as tainted. NOTE that it is very important to make the method marking the JVMInstance tainted synchronized, and to make startApplet() synchronized as well. startApplet() needs to be changed to return false if the JVM instance is tainted. recycleAppletID() is already synchronized which is needed for correctness. A synchronized query method, isTainted(), also needs to be added to JVMInstance.
7. In JVMInstance.unregisterApplet(), after all of the other existing work, if the instance is tainted and there are no other applets executing, it should send a ShutdownJVMMessage down the pipe. All other associated cleanup will be automatic.
8. In PluginMain.java, in the processing of the ShutdownJVMMessage, a check against disconnectedApplets.isEmpty() is needed. If this set is not empty, then the ShutdownJVMMessage needs to be ignored rather than calling System.exit(0).
9. In JVMManager.java, getBestJVMInstance(), the loop which checks the product version and JVMParameters needs to be changed so that it never considers a tainted JVM instance. There is an inherent race condition here, but the retry loop in JVMManager.startApplet() will handle it.

Once all of this is implemented, the necessary infrastructure will be in place for further heuristics. For example, if a signed applet loads custom native code, we might want to detect that and mark the JVM instance as tainted.