Bug ID: JDK-6455606 iCMS: Foreground collector becomes active when the coordinator is in a yield loop

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 6
6 b96Fixed

Foreground collector becomes active when the coordinator is in a yield loop.

   for (unsigned i = 0; i < CMSCoordinatorYieldSleepCount &&
                       ConcurrentMarkSweepThread::should_yield(); ++i) {
     assert(!CMSCollector::foregroundGCIsActive() , "can't become true "
           "while there are pending unsatisfied yield requests");
     os::sleep(Thread::current(), 1, false);
   }

The assertion in the code above detects this condition.
The test case and the backtraces from all the threads attached.
The command line for the test is: java -server -Xmixed -DHANGINGJAVA27879 -XX:-PrintVMOptions -Xincgc -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled gc.gctests.FinalizerGC01.FinalizerGC01

SUGGESTED FIX The fix chnages the yield loop in two ways. First, the yield loop will now exit if the foreground GC becomes active, instead of assuming that this would never be the case and assert that in the loop. This fixes the assertion that was firing, and also ensures that, when the foreground GC becomes active, the loop will exit as soon as possible (the latter maybe not a big deal, as it will happen, one hopes, infrequently). Second, the yield loop now acknowledges asynchronous yield requests after sleeping and before checking the exit condition. This means that, in the case when an asynchronous yield request comes in, the loop will actually exit only if there's still a pending synchronous request.
08-08-2006
EVALUATION Ramki spotted that there's one more issue with the yield loop. It's possible, in the case of iCMS, for the yield loop to get "stuck" if frequent async yield requests come in. This is because the loop will not exit while a yield is spending. However, the loop doesn't distinguish between synchronous and asynchronous yields. So, we should really only stay in the loop only if a synchronous yield is pending.
08-08-2006
SUGGESTED FIX The assertion failure in the test Igor added to the description is very reproducible in the original code. I can get it to crash with 2-3 iterations. The fix did 50 successful iterations with no failures.
01-08-2006
SUGGESTED FIX The fix is straightfoward: remove the assertion and add "the GC is not in the foreground" do the loop condition.
01-08-2006
EVALUATION The problem is caused by the fix for CR6442774. It turns out that it is possible for CMS to want to change from background to foreground mode while we're sleeping in the GC thread and waiting for a mutator thread to pick up the lock that it needs. We had assumed that this would not happen, and we guarded this by asserting inside the loop that the GC is not in the foreground yet. This is the assertion that fails. In the product build, which is "sans" assertions, this would not crash nor deadlock and would simply delay by a little bit the foreground GC from starting
01-08-2006