FULL PRODUCT VERSION :
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Linux 2.6.29-2-amd64 #1 SMP Sun May 17 17:15:47 UTC 2009 x86_64 GNU/Linux
Debian
A DESCRIPTION OF THE PROBLEM :
Using SoftReference objects causes the Garbage Collector to enter the worst case scenario for garbage collection.
The rules for SoftReferences when performing a full garbage collection are:
1. The garbage collector does a full mark operation over all the strongly and softly held objects, and calculates the total amount of space used (and therefore the total amount of space free).
2. The garbage collector multiplies the amount of heap free by the -XX:SoftRefLRUPolicyMSPerMB parameter (default 1000), to get a maximum soft reference age in milliseconds.
3. A second mark operation is performed over the SoftReferences, clearing all that are older than the calculated cutoff point.
4. The garbage collector continues with the sweep and compact stages.
If one writes a program that constantly generates soft referenced objects, then this scheme will work well, as at some point the amount of free space calculated will be small, and so a short maximum age will be calculated and soft references will be discarded. However, if lots of collectible objects are created as well as softly-reachable objects, then when the mark operation is concluded there will always be plenty of free space, resulting in a large calculated maximum soft reference age, and no soft references are collected.
The policy is such that the heap will fill up with soft references to such an extent that the garbage collector enters the worst case scenario of performing a full garbage collection every time, with almost no gain, and almost no interval between collections. Ideally, the garbage collector should clear soft references enough to ensure the heap has enough free space for garbage collection to be efficient.
The bug is not that the VM is not following the soft reference policy as documented, as it does. The bug is in the policy itself which causes undesirable behaviour.
Ironically, the more memory Java is allowed to use, the worse the problem.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run any program that as part of its operations creates SoftReferences and uses them to keep track of objects in a cache manner, depending on the garbage collector to retire old entries. Basically, what SoftReference is advertised to accomplish.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
I would expect the JVM to throw away soft references early enough to keep the VM running efficiently.
ACTUAL -
We regularly see our JVMs running extremely slowly, and occasionally crashing with a message saying "java.lang.OutOfMemoryError: GC overhead limit exceeded". Upon investigation, it appears that when the JVM is running slowly, it is spending almost all of its time performing full garbage collections.
In the provided test case, it would be instructive to execute with the -verbose:gc option, and with a large heap size (say 5GB), as larger heap sizes appear to make the problem worse. I also use the -server option.
The output messages (on creating every 10000 soft reference) will slow down very significantly, and you may even get an OutOfMemoryError.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
java.lang.OutOfMemoryError: GC overhead limit exceeded
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.SoftReference;
import java.util.HashSet;
public class TestSoftReferences2
{
public static void main(String args[]) {
int count = 0;
int collected = 0;
ReferenceQueue q = new ReferenceQueue();
HashSet<Reference> refs = new HashSet<Reference>();
try {
while (true) {
for (int i = 0; i < 10; i++) {
byte junk[] = new byte[1000];
}
byte lump[] = new byte[1000];
Reference ref = new SoftReference(lump, q);
refs.add(ref);
count++;
int lastCollected = collected;
Reference queued = null;
while ((queued = q.poll()) != null) {
refs.remove(queued);
collected++;
}
if (count % 10000 == 0) {
System.out.println("Created: " + count + ", collected: " + collected
+ ", active: " + (count - collected));
}
}
} finally {
System.out.println("Created: " + count + ", Collected: " + collected);
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
Reducing the -XX:SoftRefLRUPolicyMSPerMB parameter to an extremely low value (we use 1) will prevent the problem occuring, but it is not an ideal solution. It may throw away too much, and it may not be sufficient in the future with extremely large heap sizes.
The proper solution would be to make the heap free calculation be less by the size of the new generation, to ensure that soft references are always cleared in a manner that allows the collector to perform a fast garbage collection next time. Recall that the garbage collector must perform a full collection if the entire contents of (full) new generation cannot fit into the tenured generation. Keeping the size of the new generation free in the tenured generation will allow fast garbage collections to occur. Then the -XX:SoftRefLRUPolicyMSPerMB option will be a less sensitive parameter, and the system will guarantee that SoftReferences do not fill the tenured generation beyond the point at which inefficiency occurs.
An alternative solution would be for us to create a thread in our own application which has access to all the soft references, and monitors free memory, clearing soft references as necessary. However, this would be a very bad hack. The documentation for SoftReference makes it clear that this is the job of the JVM.