FULL PRODUCT VERSION :
C:\Program Files\Java\jre7\bin>.\java -version
java version "1.7.0_03"
Java(TM) SE Runtime Environment (build 1.7.0_03-b05)
Java HotSpot(TM) 64-Bit Server VM (build 22.1-b02, mixed mode)


FULL OS VERSION :
MS Windows 6.1.7601
(with Jre 7u3)
also
Linux build.tqinc.info 2.6.38-11-server #48-Ubuntu SMP Fri Jul 29 19:20:32 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
(with java 1.6.0_26)


EXTRA RELEVANT SYSTEM CONFIGURATION :
The problem does not exhibit on a 32 bit system with Windows and a less pwoerful (but still mutlithreaded - dual) processor

A DESCRIPTION OF THE PROBLEM :
The test case below sets up a class loader with some classes and objects, some with a finalize method, and nulls out all references to the whole structure and then repeatedly calls System.gc while holding a PhantomReference to one of the items to be collected.
If the code is invoked with argument 'ClassLoader' or 'Class" then all is good, and everything is collected after about 5 invocations of System.gc(). If it is 'Object' then gc fails to collect.

This test case was developed from a much more complicated situation with my product.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Yes

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
There are two source files below.
One (Main.java) is put in a directory with name memLeakTestDriver
The other (Test.java) is put in a directory with name memLeakTest

Note these names are hardcoded in the setting up of the new classloader:

for (URL u:parent.getURLs()) {
			String uu = u.toString();
			if (uu.contains("memLeakTestDriver")) {
				testCode = new URL(uu.replace("memLeakTestDriver","memLeakTest"));
			}
		}
		URLClassLoader child = URLClassLoader.newInstance(new URL[]{testCode}, parent);
		
====

Compile both classes in place, then run Main.java, e.g.
java -classpath memLeakTestDriver/bin driver.Main Object
so that the classpath substitution above works to access the Test code on the other classloader

====

If the argument to Main is 'ClassLoader' or 'Class' then the behavior is as expected, if it is 'Object' then the gc loop fails to clear the garbage.

EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected:

Hello world!
Hello world 0
Hello world 1
Hello world 2
GC 0 3
GC 1 3
GC 2 3
Bye bye world 2
GC 3 3
GC 4 2
Bye bye world 1
GC 5 2
Bye bye world 0
GC 6 1

Actual:

Hello world!
Hello world 0
Hello world 1
Hello world 2
GC 0 3
Bye bye world 2
GC 1 3
GC 2 2
GC 3 2
GC 4 2
GC 5 2
GC 6 2
GC 7 2
GC 8 2
GC 9 2
GC 10 2
GC 11 2
GC 12 2
GC 13 2
GC 14 2
GC 15 2
GC 16 2
GC 17 2
GC 18 2
GC 19 2
GC 20 2
GC 21 2
GC 22 2
GC 23 2
GC 24 2
GC 25 2
GC 26 2
GC 27 2
GC 28 2
GC 29 2
GC 30 2
GC 31 2
GC 32 2
GC 33 2
GC 34 2
GC 35 2
GC 36 2
GC 37 2
GC 38 2
GC 39 2
GC 40 2
GC 41 2
GC 42 2
GC 43 2
GC 44 2
GC 45 2
GC 46 2
GC 47 2
GC 48 2
GC 49 2
REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------

/memLeakTestDriver/src/driver/Main.java

=====

package driver;

import java.lang.ref.PhantomReference;
import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.WeakReference;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.Map;
import java.util.WeakHashMap;
import java.util.concurrent.atomic.AtomicInteger;

public class Main {

	private enum RefType {
		ClassLoader {
			@Override
			public java.lang.Object get(URLClassLoader child,
					java.lang.Class test, java.lang.Object o) {
				return child;
			}
		}, Class {
			@Override
			public java.lang.Object get(URLClassLoader child,
					java.lang.Class test, java.lang.Object o) {
				return test;
			}
		}, Object {
			@Override
			public java.lang.Object get(URLClassLoader child,
					java.lang.Class test, java.lang.Object o) {
				return o;
			}
		};
		abstract public Object get(URLClassLoader child, Class test, Object o);
	}

	public static void main(String[] args) throws Exception {
		RefType refType = null;
		try {
			refType = RefType.valueOf(args[0]);
			
		}
		catch (Exception e) {
			System.err.println("Expected argument: 'ClassLoader' or 'Class' or 'Object'");
			return;
		}
		

		
		URLClassLoader parent = (URLClassLoader) Main.class.getClassLoader();
		URL testCode = null;
		for (URL u:parent.getURLs()) {
			String uu = u.toString();
			if (uu.contains("memLeakTestDriver")) {
				testCode = new URL(uu.replace("memLeakTestDriver","memLeakTest"));
			}
		}
		URLClassLoader child = URLClassLoader.newInstance(new URL[]{testCode}, parent);
		Class test = child.loadClass("test.Test");
		Object o = test.newInstance();
		Map<ClassLoader,Reference<Class>> weak = new WeakHashMap<ClassLoader,Reference<Class>>();
		weak.put(child, new WeakReference(test));

		o = refType.get(child, test, o);
		
		ReferenceQueue queue = new ReferenceQueue();
		PhantomReference ref = new PhantomReference(
				o, // can be changed to 'test' or 'child' and problem goes away
				queue);
		AtomicInteger problemCount = (AtomicInteger) test.getDeclaredMethod("getProblemCount").invoke(null);
		o = null;
		test = null;
		child = null;
		int cnt = 0;
		while (problemCount.get() != 0 && cnt < 50 ) {
			//	queue.poll()==null && cnt < 30) {
			System.gc();
			System.err.println("GC "+cnt+" "+problemCount.get());
			cnt++;
		}
		ref.toString();
		
	}

}


============================
/memLeakTest/src/test/Test.java
============================
package test;

import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;

public class Test {
	
    static Object problem;
    
    static public AtomicInteger getProblemCount() {
    	return Problem.aliveCount;
    }
	
	public Test()
	{
		System.err.println("Hello world!");
		problem = new Problem();
		new Problem();
	}

}

class Problem {
	static AtomicInteger aliveCount = new AtomicInteger(0);
	
	static Problem test = new Problem();
	final int id = aliveCount.getAndIncrement();
	
	public Problem()  {
		System.err.println("Hello world "+id);
	}

	@Override
	protected void finalize() {
		System.err.println("Bye bye world "+id);
		aliveCount.decrementAndGet();
	}
}


---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Manually null out some of the suspect references