JDK-6392332 : Handling of SIGTERM is slow under heavy GC load
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 6
  • Priority: P3
  • Status: Closed
  • Resolution: Not an Issue
  • OS: solaris
  • CPU: x86
  • Submitted: 2006-03-01
  • Updated: 2012-02-01
  • Resolved: 2006-03-17
Related Reports
Relates :  
JDK-4073195 - (process) Process.destroy() isn't guaranteed to kill child process on Solaris
Description
Problem happens when java test that stresses GC receives SIGTERM. It may take long time to exit after test process receives SIGTERM from harness(which is using Process.destroy()). It may be also reproduced manually by running the test and pressing Ctrl-C. During my manual experiments, it took up to 15 minutes. However, during nightly testing, many test processes are left for longer time. It seems that when there are multiple java processes handling SIGTERM, the time may be much longer.
To reproduce it manually:
ssh to solaris-x86 machine (I reproduced it on vm-linux-1.sfbay)
/net/gtee.sfbay/export/nightly/mustang/JDK/gc_baseline/jdk1.6/solaris-i586/bin/java -cp /net/gtee.sfbay/export/gtee/suites/testbase_vm.1.6/vm/bin/classes -server -Xmixed -DHANGINGJAVA13719 -XX:-PrintVMOptions -XX:+UseParallelGC -XX:+UseParallelOldGC -Xms120m gc.gctests.FinalizeTest02.FinalizeTest02
(Press Ctrl-C and observe what happens. Repeated Ctrl-C does not seem to help)

I tried to use jstack on the process after hitting Ctrl-C, but it seems to also hang. Pstack works, several different native thread dumps are attached to this bug report.

This problem severly affects nightly testing: harness cannot timely terminate child test processes and they fill all available memory on the machine, causing unexpected failures.
Comments
EVALUATION In short there is no defect here. An orderly shutdown requires processing that is concurrent with the other activities of the VM/application and can be arbitrarily delayed, or even not occur at all, due to resource limitations, including lack of CPU time or insufficient memory. If you require a more immediate, and abrupt, shutdown of the VM then you must use an alternate mechanism than sending SIGTERM.
17-03-2006
EVALUATION Under Solaris and Linux the signals SIGHUP, SIGINT and SIGTERM all initiate an orderly shutdown of the VM, unless the -X:rs option is specified to reduce signal usage. An orderly shutdown by one of these signals involves running all registered shutdown hook threads. The initial stage of signal handling requires a call of the Java sun.misc.Signal.dispatch method, which then creates and starts a new Java thread to invoked the registered SignalHandler (if there is one). If there is heavy GC activity and limited available memory then this process will be delayed. Even though the created signal handler thread runs with "nearMax" priority that may be of little help if any actual shutdown hook threads must be executed. If there is insufficient memory to even start the new thread then the shutdown process is not even initiated (and a warning is printed to the console.) and so a call to Process.destroy would have no effect at all.
02-03-2006