FULL PRODUCT VERSION :
java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java Hotspot(TM) Client VM (build 10.0-b22, mixed mode, sharing)

java version "1.6.0_10"
Java(TM) SE Runtime Environment (build 1.6.0_10-b33)
Java HotSpot(TM) Client VM (build 11.0-b15, mixed mode, sharing)


ADDITIONAL OS VERSION INFORMATION :
Linux 2.6.13-15.8-smp x86_64

EXTRA RELEVANT SYSTEM CONFIGURATION :
Reproduce with Mozilla 1.7.11 and Firefox 1.0.6.
My colleages reproduced with Firefox 2 and Firefox 3.

A DESCRIPTION OF THE PROBLEM :
If I open the web browser (firefox or mozilla) and load Java applets one by one, it eventually runs out of memory (PermGen space). The reason is that it does not release old classloaders of applets that are not anymore active, and hence does not release the classes that were allocated by this classloader.

If I have more than 5 applets and browse them one by one,  and after the last applet I come back to the first applet, I see that its classes are reloaded again by a new classloader, even though the classes of this applet are still in the VM hold by an old classloader.

The increase of the code memory can easily be observed by using jconsole.

I analyzed the situation with jhat, and it seems all class loaders are kept in a HashMap in a static field of sun.awt.X11.XToolkit.winToDispatcher. This is of course a memory leak, since when the class loader is hold, all applet classes loaded by the classloader are also hold and never freed by the Garbage collection.

The exact reference chain (shown by jhat) is this:

Static reference from sun.awt.X11.XToolkit.winToDispatcher (from class sun.awt.X11.XToolkit) :
--> java.util.HashMap@0x79d7ce38 (40 bytes) (field table:)
--> [Ljava.util.HashMap$Entry;@0x79d98670 (72 bytes) (Element 8 of [Ljava.util.HashMap$Entry;@0x79d98670:)
--> java.util.HashMap$Entry@0x79e75490 (24 bytes) (field value:)
--> java.util.Vector@0x79e76220 (24 bytes) (field elementData:)
--> [Ljava.lang.Object;@0x79e77450 (48 bytes) (Element 0 of [Ljava.lang.Object;@0x79e77450:)
--> sun.awt.X11.XEmbedClientHelper@0x79e77438 (22 bytes) (field embedded:)
--> sun.awt.X11.XEmbeddedFramePeer@0x79e760b8 (301 bytes) (field target:)
--> sun.plugin.viewer.frame.XNetscapeEmbeddedFrame@0x79e75618 (361 bytes) (field appContext:)
--> sun.awt.AppContext@0x79e755e0 (49 bytes) (field contextClassLoader:)
--> sun.plugin.security.PluginClassLoader@0x79e74ac0 (123 bytes)

All class loaders are hold by such a chain.

Indeed the XEmbedClientHelper.install() adds itself to the XToolkit but never removes itself. I think it should remove itself when the applet is destroyed. The XEmbedClientHelper has a link chain to the AppContext, which holds the class loader.




STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
In principle, it can be reproduced by creating a couple of large applets (with lots of classes) and load them one by one into the webbrowser without shutting the webbrowser down. The increase in code can be watched with jconsole. If there are many applets, you will eventually notice the OutOfMemory (PermGen space) no matter how high your PermGenSize setting is.

For your conveniance, here a csh script that generated 6 large applets,
each with 1000 classes:

---------------------------- cut -------------------------------------------
#!/bin/csh

mkdir a
mkdir b
mkdir c
mkdir d
mkdir e
mkdir f

foreach d (a b c d e f)

	pushd .
	cd $d
	touch MyApplet.java
	echo "package "$d";" >> MyApplet.java
	echo "import java.awt.*;" >> MyApplet.java
	echo "import java.awt.event.*;" >> MyApplet.java
	echo "import javax.swing.*;" >> MyApplet.java
	echo "public class MyApplet extends JApplet {" >> MyApplet.java
	echo "  public void init() {" >> MyApplet.java
	echo "    super.init();" >> MyApplet.java
	echo '    JButton helpButton = new JButton("Test");' >> MyApplet.java
	echo "    helpButton.addActionListener(new ActionListener() {" >> MyApplet.java
	echo "      public void actionPerformed(ActionEvent evt) {" >> MyApplet.java
	echo '        System.err.println("Help pressed"); ' >> MyApplet.java
	echo "      }});" >> MyApplet.java
	echo "    JPanel panel = new JPanel(new FlowLayout());" >> MyApplet.java
	echo "    panel.add(helpButton);" >> MyApplet.java
	echo "    getContentPane().add(panel);" >> MyApplet.java

	foreach i (0 1 2 3 4 5 6 7 8 9)
	foreach j (0 1 2 3 4 5 6 7 8 9)
	foreach k (0 1 2 3 4 5 6 7 8 9)
	        echo "    new Class"$i$j$k"();" >> MyApplet.java
		touch Class$i$j$k.java
		echo "package "$d";" >> Class$i$j$k.java
		echo "public class Class"$i$j$k >> Class$i$j$k.java
		echo "{" >> Class$i$j$k.java
		foreach m (0 1 2 3 4 5 6 7 8 9)
	        foreach n (0 1 2 3 4 5 6 7 8 9)
			echo "  public void method"$m$n"() {}" >> Class$i$j$k.java
		end
		end
		echo "}" >> Class$i$j$k.java
	end
	end
	end

	echo "  }" >> MyApplet.java
	echo "}" >> MyApplet.java
	popd

end
--------------------- cut ----------------------------------------

Run this script in Linux or Unix creates directories a b c d e f which contains the applet 6 times. The only difference is that package name.
Compile these one by one and each package into one jar:

javac -d . src/a/*.java
javac -d . src/b/*.java
javac -d . src/c/*.java
javac -d . src/d/*.java
javac -d . src/e/*.java
javac -d . src/f/*.java
jar cf appletA.jar a/
jar cf appletB.jar b/
jar cf appletC.jar c/
jar cf appletD.jar d/
jar cf appletE.jar e/
jar cf appletF.jar f/

Now you have 6 applets, each just containing one button.
The applet creates instances of 1000 classes, just to make sure 1000 classes are loaded. The instances are garbage collectable.

Now create 6 HTML files that load the applets.
Here is one (index0.html):

----------------- cut here -------------------------------------------------
<HTML>
<HEAD>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<TITLE>Test1</TITLE>
</HEAD>
<BODY bgcolor="#FFFFFF">
<!--"CONVERTED_APPLET"-->
<!-- HTML CONVERTER -->
<OBJECT
    classid = "clsid:8AD9C840-044E-11D1-B3E9-00805F499D93"
    codebase = "http://java.sun.com/update/1.4.2/jinstall-1_4-windows-i586.cab#Version=1,4,0,0"
    WIDTH = "300" HEIGHT = "40" ALIGN = "baseline" >
    <PARAM NAME = CODE VALUE = "a.MyApplet" >
    <PARAM NAME = CODEBASE VALUE = "." >
    <PARAM NAME = ARCHIVE VALUE = "appletA.jar" >
    <PARAM NAME = "type" VALUE = "application/x-java-applet;version=1.4">
    <PARAM NAME = "scriptable" VALUE = "false">

    <COMMENT>
	<EMBED
            type = "application/x-java-applet;version=1.4"             CODE = "a.MyApplet"             JAVA_CODEBASE = "."             ARCHIVE = "appletA.jar"             WIDTH = "300"             HEIGHT = "40"             ALIGN = "baseline" 	    scriptable = false 	    pluginspage = "http://java.sun.com/products/plugin/index.html#download">
	    <NOEMBED>
            
            </NOEMBED>
	</EMBED>
    </COMMENT>
</OBJECT>

<!--
<APPLET CODE = "a.MyApplet" JAVA_CODEBASE = "." ARCHIVE = "appletA.jar" WIDTH = "300" HEIGHT = "40" ALIGN = "baseline">


</APPLET>
-->


<!--"END_CONVERTED_APPLET"-->
<!--@@@-->

<p>

<A HREF="index1.html">NEXT</A>

</BODY>
</HTML>
----------------------------------- cut here -------------------------------

Create similar index1.html, index2.html ... index5.html.
Each one should have the NEXT link pointing to the next one,
and index5.html should have the NEXT link pointing to index0.html.
Each should load a different applet, i.e.
index1.html loads b.MyApplet  in appletB.jar
index2.html loads c.MyApplet in appletC.jar
...
index5.html loads f.MyApplet in appletF.jar.


Now open index0.html, wait until the applet is loaded,
then repeat to click NEXT, each click loads a new applet,
until OutOfMemory. If you don't want to wait until OutOfMemory,
use jconsole to see the code memory increase for each applet
that is loaded.

When I tried this with only 3 applets, the classLoaderCache
of the plugin avoided that the classes of an applet are reloaded.
In this case, hit the x key in the Java Console before clicking the
NEXT link to force that an previously loaded applet is reloaded.

On my machine, with 6 applets, when coming from the last applet
to the first applet, it is reloaded independent of hitting the x key.



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
When an applet is destroyed, then all its memory should be
freed. The applet classloader should not be hold by a reference
chain from XToolkit.

Or to say it differently:
If I have 100 large applets that individually are small enough to
run in the brower without OutOfMemory, then it should be possible
to browse through all 100 large applets without shutting down
the browser and without OutOfMemory caused by accumulated
memory.

ACTUAL -
OutOfMemory (PermGen space)

ERROR MESSAGES/STACK TRACES THAT OCCUR :
OutOfMemory (PermGen space)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
See Description. It contains a csh script to generate 6 large applets.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Shutting down the browser before loading the next applet avoids the OutOfMemory.