FULL PRODUCT VERSION :
java version "1.5.0_04"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode, sharing)

ADDITIONAL OS VERSION INFORMATION :
Linux amontillado 2.6.8-2-686 #1 Tue Aug 16 13:22:48 UTC 2005 i686 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
Shortly after starting a Swing application it completely freezes up.  A thread dump shows that the application is deadlocked due to locks in the AppClassLoader and the ExtClassLoader.  The thread dump is included below.
Just to give you a   little more info about what I think is causing the problem.. We are   using the "-Djava.ext.dirs" property to load application JAR files.   
Something like: java -Djava.ext.dirs=our-lib-dir MainClass

Where our-lib-dir/ has JAR files containing our application code as   well as third-party library JARs.  From everything I understand about the extension mechanism this is incorrect, but it's easy, so someone decided to go that route.  We're fixing that.

I think this might be contributing to the problem because the deserialization code is using the ExtClassLoader to load our application classes at the same time a Java platform class is being loaded from the ExtClassLoader.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
I've been able to duplicate the problem using a debugger to control program flow. Ideally a deadlock should never occur under *any* circumstances, so if I can demonstrate it with a debugger, then it's a valid bug.  That was my thinking, anyhow.

The basic idea is to create a JAR file with an application class that launches two threads: one that causes some XML classes to be loaded, the other triggers a class load from a JAR file in the extension directory.  After launching both threads, we want to pause them at appropriate points to get them to acquire mutually exclusive locks.

I'm using JDK 1.5.0_04 for the following.

1. Run CreateSerialized to create a file (object.tmp) containing a serialized instance of a InternetAddress from the JavaMail API.  I  chose this class for a couple reasons.  One, it is contained inside a JAR file that is encrypted, so the encryption code will run just like in the original stacktrace.  Two, it is publicly available.  And three, it is not in the application JAR file (not sure if this makes  
a difference).

2. Compile Deadlocker and place it's classes into a JAR file.  Place this JAR file into testlib/

3. Place mail.jar into testlib/

4. Load Deadlocker into a debugger.  I used Intellij IDEA.  The debugger must support thread-level breakpoints.  Place the following  *thread-level* breakpoints:

javax.xml.parsers.FactoryFinder.newInstance line 88 ("providerClass =  
cl.loadClass(className)")
Deadlocker line 32 ("ObjectInputStream objectInputStream = new  
ObjectInputStream(new ByteArrayInputStream(bytes));")

4. Run Deadlocker like so:

java -Xdebug -Xnoagent -Djava.compiler=NONE - 
Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 - 
Djava.ext.dirs=testlib Deadlocker

5. Deadlocker will pause for 10 secs on startup. During this time connect to it with the debugger.  After 10 secs the first two breakpoints will be hit.

6. Go to the breakpoint in FactoryFinder on the "XML" thread and step into the loadClass method, step into the ClassLoader#loadClass method, and keep stepping until the "XML" thread has locked the AppClassLoader.  I had to step until I got to the System#getSecurityManager method to make this happen.  (Triggering thread dumps with each step to inspect the locks helps to tell when the lock has been acquired).  Leave this thread alone for the moment.

7. Go to the breakpoint in Deadlocker on the "Read Object" thread and resume that thread.  It should acquire a lock on the ExtClassLoader, then wait on the AppClassLoader.

8. Return to the "XML" thread and resume it.  At this point a thread dump should show one deadlock between the two threads.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Found one Java-level deadlock:
=============================
"com.reflexsecurity.common.utils.io.FragmentingTransactionInputEngine":
  waiting to lock monitor 0x0809c814 (object 0x5ad09198, a
sun.misc.Launcher$AppClassLoader),
  which is held by "AWT-EventQueue-0"
"AWT-EventQueue-0":
  waiting to lock monitor 0x0809c5d4 (object 0x5ad08f68, a
sun.misc.Launcher$ExtClassLoader),
  which is held by
"com.reflexsecurity.common.utils.io.FragmentingTransactionInputEngine"

Java stack information for the threads listed above:
===================================================
"com.reflexsecurity.common.utils.io.FragmentingTransactionInputEngine":
        at sun.security.jca.ProviderConfig.getProvider(Unknown Source)
        - waiting to lock <0x5ad09198> (a sun.misc.Launcher$AppClassLoader)
        at sun.security.jca.ProviderList.getProvider(Unknown Source)
        at sun.security.jca.ProviderList$ServiceList.tryGet(Unknown Source)
        at sun.security.jca.ProviderList$ServiceList.access$200(Unknown Source)
        at sun.security.jca.ProviderList$ServiceList$1.hasNext(Unknown Source)
        at java.security.KeyFactory.nextSpi(Unknown Source)
        - locked <0x5992c620> (a java.lang.Object)
        at java.security.KeyFactory.<init>(Unknown Source)
        at java.security.KeyFactory.getInstance(Unknown Source)
        at sun.security.x509.X509Key.buildX509Key(Unknown Source)
        at sun.security.x509.X509Key.parse(Unknown Source)
        at sun.security.x509.CertificateX509Key.<init>(Unknown Source)
        at sun.security.x509.X509CertInfo.parse(Unknown Source)
        at sun.security.x509.X509CertInfo.<init>(Unknown Source)
        at sun.security.x509.X509CertImpl.parse(Unknown Source)
        at sun.security.x509.X509CertImpl.<init>(Unknown Source)
        at sun.security.provider.X509Factory.engineGenerateCertificate(Unknown
Source)
        at java.security.cert.CertificateFactory.generateCertificate(Unknown Source)
        at sun.security.pkcs.PKCS7.parseSignedData(Unknown Source)
        at sun.security.pkcs.PKCS7.parse(Unknown Source)
        at sun.security.pkcs.PKCS7.parse(Unknown Source)
        at sun.security.pkcs.PKCS7.<init>(Unknown Source)
        at sun.security.util.SignatureFileVerifier.<init>(Unknown Source)
        at java.util.jar.JarVerifier.processEntry(Unknown Source)
        at java.util.jar.JarVerifier.update(Unknown Source)
        at java.util.jar.JarFile.initializeVerifier(Unknown Source)
        at java.util.jar.JarFile.getInputStream(Unknown Source)
        - locked <0x5ad490a8> (a java.util.jar.JarFile)
        at sun.misc.URLClassPath$JarLoader$1.getInputStream(Unknown Source)
        at sun.misc.Resource.cachedInputStream(Unknown Source)
        - locked <0x598895b0> (a sun.misc.URLClassPath$JarLoader$1)
        at sun.misc.Resource.getByteBuffer(Unknown Source)
        at java.net.URLClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.access$100(Unknown Source)
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        - locked <0x5ad08f68> (a sun.misc.Launcher$ExtClassLoader)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClassInternal(Unknown Source)
        - locked <0x5ad08f68> (a sun.misc.Launcher$ExtClassLoader)
        at java.lang.Class.getDeclaredFields0(Native Method)
        at java.lang.Class.privateGetDeclaredFields(Unknown Source)
        at java.lang.Class.getDeclaredField(Unknown Source)
        at java.io.ObjectStreamClass.getDeclaredSUID(Unknown Source)
        at java.io.ObjectStreamClass.access$600(Unknown Source)
        at java.io.ObjectStreamClass$2.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.io.ObjectStreamClass.<init>(Unknown Source)
        at java.io.ObjectStreamClass.lookup(Unknown Source)
        at java.io.ObjectStreamClass.initNonProxy(Unknown Source)
        at java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
        at java.io.ObjectInputStream.readClassDesc(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
......
Found 1 deadlock.

REPRODUCIBILITY :
This bug can be reproduced rarely.

---------- BEGIN SOURCE ----------

import java.io.FileOutputStream;
import java.io.ObjectOutputStream;
import javax.mail.internet.InternetAddress;

public class CreateSerialized {
	public static void main(String[] args) throws Exception {
		Object o = InternetAddress.parse("###@###.###")[0];

		FileOutputStream fileOutputStream = new FileOutputStream 
("object.tmp");
		ObjectOutputStream objectOutputStream = new ObjectOutputStream 
(fileOutputStream);
		objectOutputStream.writeObject(o);
		fileOutputStream.close();
	}
}

import java.io.File;
import java.io.FileInputStream;
import java.io.ObjectInputStream;
import java.io.ByteArrayInputStream;
import javax.xml.parsers.DocumentBuilderFactory;

public class Deadlocker {
	public static void main(String[] args) throws Exception {
		File file = new File("object.tmp");
		final byte[] bytes = new byte[(int)file.length()];
		FileInputStream fileInputStream = new FileInputStream("object.tmp");
		int read = fileInputStream.read(bytes);
		if (read != file.length()) {
			throw new Exception("Didn't read all");
		}

		Thread.sleep(10000);

		Runnable xmlRunnable = new Runnable() {
			public void run() {
				try {
					DocumentBuilderFactory.newInstance();
				} catch (Exception e) {
					e.printStackTrace();
				}
			}
		};

		Runnable readObjectRunnable = new Runnable() {
			public void run() {
				try {
					ObjectInputStream objectInputStream = new ObjectInputStream(new  
ByteArrayInputStream(bytes));
					Object o = objectInputStream.readObject();
					System.out.println(o.getClass());
				} catch (Exception e) {
					e.printStackTrace();
				}
			}
		};

		Thread thread1 = new Thread(readObjectRunnable, "Read Object");
		Thread thread2 = new Thread(xmlRunnable, "XML");

		thread1.start();
		thread2.start();

		thread1.join();
		thread2.join();
	}
}

---------- END SOURCE ----------