Windows loads dll's in various awkward places in the 32-bit virtual
address space, leading to virtual address space fragmentation. The
jvm requires contiguous virtual address space for the heap, so if
many dll's have been loaded, it's quite likely that for a given
on-the-large-size value of -Xmx, the jvm may not be able to obtain
a large enough contiguous memory region for the heap.
There have been a large number of negative comments regarding the above evaluation and the decision to close this bug as "will not fix".
Up until recently, it was not technically feasible to fix this problem in the Java Plug-In due to its in-process architecture.
In recent months a new implementation of the Java Plug-In has been developed which changes the execution model of applets. Instead of running the applets in a JVM embedded in the web browser's process, they are now run in a separate JVM process which communicates back to the web browser. This work was checked in under 6622332.
This new plug-in solves this problem in two ways. First, since the JVM is no longer running in the same process as the web browser, the address space fragmentation problem in IE described above does not interfere with the specification of a large -Xmx value (for example, in the Java Runtime Parameters in the Java Control Panel).
Second, the new Java Plug-In will support specification of JVM command-line arguments on a per-applet basis via a new parameter in the applet's HTML. This will eliminate the need to do any global setting of this or other parameters in the Java Control Panel.
The first public release of the new Java Plug-In will be in build 08 of the forthcoming "6uN" release, for which early access builds can be found at https://jdk6.dev.java.net/6uNea.html. As of this writing the new Java Plug-In is not yet the default, but users will be able to turn it on by toggling a Java Control Panel switch. More documentation on the new plug-in will be provided when it becomes available or soon afterward.
To indicate that we have heard your feedback and have responded, the disposition of this bug is being changed from "will not fix". Instead, for tracking purposes, it is being closed as a duplicate of 6622332. We hope that the new plug-in will address this significant issue.
Note that currently the new plug-in does not change the default heap size for applets, but we believe that this will be far less of an issue now that an individual applet can request a particular heap size. We are open to revisiting this particular issue, in particular in the context of the new plug-in.
Please try the new Java Plug-In when it becomes available and file any issues against it under product java, category java_plugin, subcategory plugin2.