The new plugin has different behavior than the old plugin while starting applets. The old plugin used to initialize applets serially and block the browser while doing so. The new plugin does not block the browser; applets are started asynchronously. Additionally, there is no class loader and AppContext caching in the new plugin (currently), so each of the six applet instances on the web page will be loaded in its own AppContext and receive its own AWT Event Dispatch Thread, meaning that repainting of the applets will occur in parallel whereas before they used to be repainted serially.
These differences in behavior basically mean that this particular test case requires roughly six times the RAM it required previously.
Attempting to increase the heap size using the Java Control Panel uncovered several bugs in the management of the JVMParameters data structures used to keep track of which attached JVM instance is suitable for running a particular applet. These bugs as well as others have been fixed. The test case will now run correctly if for example -Xmx512m is specified in the Java Applet Runtime Parameters in the Java Control Panel.