JDK-7069418 : Cancelled tasks prevent ScheduledThreadPoolExecutor from shutting down
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util.concurrent
  • Affected Version: 7
  • Priority: P3
  • Status: Closed
  • Resolution: Won't Fix
  • OS: windows_7
  • CPU: x86
  • Submitted: 2011-07-21
  • Updated: 2012-03-20
  • Resolved: 2011-08-16
Related Reports
Relates :  
Description
FULL PRODUCT VERSION :
java version "1.7.0"
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Java HotSpot(TM) Client VM (build 21.0-b17, mixed mode, sharing)


ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
ExecutorService.awaitTermination() is supposed to block "until all tasks have completed execution after a shutdown request" yet the testcase demonstrates a case where this is not true (the executor blocks forever).

Proposed fix: ScheduledThreadPoolExecutor.ScheduledFutureTask.cancel() should fire ThreadPoolExecutor.interruptIdleWorkers()

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run testcase

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The executor should shut down once the scheduled task is cancelled.
ACTUAL -
The executor blocks forever.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------

Enable ScheduledThreadPoolExecutor.setRemoveOnCancelPolicy() but this was only added in JDK 7public class Testcase
{
	public static void main(String[] args) throws InterruptedException
	{
		ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
		ScheduledFuture<?> future = executor.schedule(new Callable<Void>()
		{
			@Override
			public Void call() throws Exception
			{
				System.out.println("Task ran!");
				return null;
			}
		}, 1, TimeUnit.DAYS);
		
		executor.shutdown();
		boolean cancelled = future.cancel(false);

		// future.cancel() returns success
		assert (cancelled);
		if (cancelled)
			System.out.println("Task successfully cancelled");
		else
			System.out.println("Task cannot be cancelled");
		try
		{
			// The worker thread prevents the executor from shutting down even though all tasks are
			// cancelled.
			System.out.println("Waiting for executor to shut down...");
			executor.awaitTermination(Long.MAX_VALUE, TimeUnit.DAYS);
			System.out.println("Executor successfully shut down...");
		}
		catch (InterruptedException e)
		{
			e.printStackTrace();
		}
	}
}

Comments
EVALUATION As David says, this is fixed as much as it likely to be fixed in JDK 7 so we are closing this bug.
16-08-2011

EVALUATION Doug Lea adds: If we did this, there would be at least one visible difference: A shutdown followed by cancel followed by shutdownNow would no longer contain the task in the list returned by shutdownNow. I can imagine someone out there relying on this. Supporting all the different policy options doesn't give us a lot of room to change code to deal with this case. I don't see anything we could change that wouldn't lead to someone else having some different failed expectation.
29-07-2011

EVALUATION The fact that onShutdown() happens to fix the submitters problem does not mean that it would be appropriate to invoke it when cancelling all tasks. What the submitter is relying on from onShutdown() is that when the policy is to continue to execute periodic tasks after shutdown, onShutdown() just happens to include cancelled tasks in the set of tasks to be removed. In effect the submitter is trying to implement a per-task remove-on-cancel policy, but unfortunately because the type of a task is opaque there is no place for such a policy to be defined. (It would have to be part of an exposed ScheduledFutureTask type, or added to a new interface for tasks submitted to executors [which doesn't exist].) Consequently the only option is to have a per-executor policy for remove-on-cancel which is exactly what was added for Java 7.
29-07-2011

EVALUATION Upon futher investigation that submitter have noticed that invoking onShutdown() in ScheduledFutureTask.cancel() fixes the problem. So he propose adding the following line of code to the end of ScheduledFutureTask.cancel(): if(isShutdown()) onShutdown(); ScheduledThreadPoolExecutor.setContinueExistingPeriodicTasksAfterShutdownPolicy() already does this and it seems to work without the need to lock the queue.
27-07-2011

EVALUATION The original design and implementation of ScheduledThreadPoolExecutor (STPE) made some assumptions about how STPE would be used. Part of these assumptions concerned the cancellation policies for submitted tasks. While it might seem "natural" to expect a cancelled task to be removed from the STPE queue, the implementation would require a linear search of the internal PriorityQueue implementation, during which the queue would be locked. This made removal on cancellation a very expensive operation. Consequently it was not made the policy to remove cancelled tasks. Instead the implementation relied on two other aspects of the design to manage cancelled tasks in a timely manner: a) cancelled tasks would be removed as they fell due for execution; and b) the ThreadPoolExecutor (TPE) purge operation could be used to (periodically) expunge cancelled tasks from the queue As experience with using STPE grew it was found that some use-cases created many, many cancelled tasks which consumed large amounts of memory and also potentially caused undesirable "stalls" as the queue was purged. To acommodate these scenarios it was decided to add a policy to STPE that enabled remove-on-cancel (disabled by default), in combination with reworking the internal queue implementation to make removal less expensive. This API was added in 2007, under CR 6602600 for Java 7. Unfortunately Java 7 is only now being released. Also in Java 6, additional methods were added to TPE/STPE that allowed you to create a subclass with more control over the decorator classes used to wrap the Runnable/Callable objects submitted to the TPE/STPE. Custom decorator classes can then allow you to utilize the TPE.remove(task) operation for direct removal of the task (where the argument to remove has to be the wrapper instance not the original Runnable/callable. These various facilities do not provide complete control over cancellation behaviour in every possible scenario, but they do cover a lot of options - particularly when coupled with the additional policy options to not execute delayed and/or periodic tasks after shutdown has been requested. Note: the suggested fix of calling interruptIdleWorkers() does not actually work. If the worker is interrupted during getTask() it will see that shutdown has been initiated but that the work queue is not empty and so it will simply loop and call getTask again - blocking on the same cancelled task.
25-07-2011

WORK AROUND There are a number of other possible workarounds: a) Modify the shutdown policies: executor.setExecuteExistingDelayedTasksAfterShutdownPolicy(false); executor.setContinueExistingPeriodicTasksAfterShutdownPolicy(false); this will clear all queued tasks at shutdown (and cancel them in the process).Of course this assumes that you don't want to continue with any existing delayed and/or periodic tasks (in which case you may as well use shutdownNow() instead of shutdown()). b) Cancel the task before doing the shutdown if possible This depends on the detailed application logic of course. It all depends on why you are shutting down and why you are cancelling. c) Use ThreadPoolExecutor.purge() to remove cancelled tasks after shutdown This requires that you have an actual ScheduledThreadPoolExecutor instance not an arbitrary ScheduledExecutorService. If a reference to the STPE is available at the point where the task is cancelled then invoke step.purge() afterwards. Otherwise you could employ this in the shutdown sequence by doing something like: executor.shutdown(); if( !executor.awaitTermination(5, TimeUnit.SECONDS)) executor.purge(); d) Use your own ScheduledExecutorService implementation, using the OpenJDK source for STPE to add removal-on-cancellation
25-07-2011

WORK AROUND Enable ScheduledThreadPoolExecutor.setRemoveOnCancelPolicy() but this was only added in JDK 7
24-07-2011