Bug ID: JDK-8290036 Define and specify Runtime shutdown sequence

JDK 20
20 b19Fixed

Changeset: d4142d84 Author: Stuart Marks <smarks@openjdk.org> Date: 2022-10-05 23:43:02 +0000 URL: https://git.openjdk.org/jdk/commit/d4142d8441172fc54c9abf0a735c30b0ac8638c5
05-10-2022
I talked to Alan about this today; he suggested a bunch of minor changes. A significant point we decided on was how to deal with an explicitly-started shutdown hook. See my comment above from 2022-07-18 with alternatives (0) through (5). Alan and I decided on (1) -- explicitly say it's unspecified -- because: we didn't want to commit to a specific behavior; doing this is an error anyway; it's long been allowed and behaved in a mostly broken way; and historically its behavior has never been specified. Note that the behavior changed in JDK 19 with the integration of Loom (Preview). Previously, the shutdown hook code would attempt to start the shutdown hooks but get an exception when it encountered one that was already started. This would occur possibly after some but not necessarily all shutdown hooks were started, and it would not wait for any hooks to terminate. The exception would be caught by the caller, who behaved as if the shutdown sequence had completed, and then proceed to terminate the VM. The new JDK 19 behavior ignores exceptions from starting an already-started hook, joins all hooks (including those explicitly started), and returns when they've all terminated.
04-08-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/9437 Date: 2022-07-08 23:00:15 +0000
03-08-2022
I think (3). By definition only an unstarted thread can potentially be a shutdown hook, so if someone starts a thread registered as a shutdown hook, then it should no longer be considered a shutdown hook. We don't need to go out of our way to detect this early and remove it when we can just detect it when we try to start() it, and thereafter ignore it as it is not in fact a shutdown hook after all.
19-07-2022
One remaining issue is how to handle the situation where a shutdown hook is or has been started explicitly by the programmer prior to the shutdown sequence. I don't have any strong opinions on this, but I do think we should decide what (if anything) to do about it in the spec. It's fairly clear to me that adding a thread as a shutdown hook and then starting it with an explicit call to start() is a programming error of some sort. However, it might be that some programs are depending on this. Here are some alternatives. 0. Do nothing. The spec leaves this case implicitly unspecified. 1. Explicitly specify that the behavior in this case is unspecified. 2. Specify something like the current JDK 19 behavior. (I think it's clear that the earlier behavior of effectively abandoning the shutdown sequence and halting immediately is undesirable. I believe it was probably an implementation oversight.) That is, it's possible call start() on a shutdown hook, if it's non-daemon it's counted along with other non-daemon threads, and the shutdown sequence joins it as if it had been started as a shutdown hook. 3. A variation of (2) but since the hook wasn't started by the shutdown sequence, it's not joined by the shutdown sequence. It's thus treated similar to any daemon or non-daemon thread, though it remains registered as a shutdown hook, which is a bit odd. 4. Calling start() on a registered shutdown hook throws an exception. Somewhat cleaner though possibly a behavior change that could be disruptive. 5. Calling start() on a registered shutdown hook starts the thread and implicitly de-registers it as a shutdown hook. Somewhat cleaner, as calling start() kind of converts a shutdown hook back into an ordinary thread.
18-07-2022
Pushed another round of updates to https://github.com/openjdk/jdk/pull/9437
18-07-2022
I've updated the draft PR: https://github.com/openjdk/jdk/pull/9437 Mainly this defines "shutdown sequence" in the Runtime class spec and centralizes the discussion there. Specs of various individual methods such as exit, halt, add/removeShutdownHook are adjusted accordingly, as is System.exit. I've not mentioned JNI DestroyJavaVM as I don't think it's necessary, per discussion in JDK-8290196. That's what initiates shutdown when the non-daemon thread count drops to zero, so I specified that instead of talking about JNI. This does NOT address the issue discussed in prior comments regarding shutdown hooks that have already been started prior the beginning of the shutdown sequence. I think we need to decide whether we want to change the behavior further, and specify that, or just leave it unspecified, or something. Don't take the exact words too seriously at the moment. Additional edits are undoubtedly necessary. In particular, the terminology used here will need to be adjusted to match terminology used in the changes to JLS 12.8 and JVMS 5.7.
15-07-2022
As you say, this is can of worms. The spec is that the VM halts after the hooks have finished. If the user has messed up, then the historical behavior was to silently shallow the IllegalThreadStateException and halt without joining/waiting for any hooks. So it may have been fail-fast but it was done silently.
14-07-2022
[~alanb] There is a general concurrency rule of thumb that you should never join() a thread you didn't start - as it usually means you have no idea under what conditions it will actually terminate. If someone directly started a hook thread, virtual or otherwise, then that is a major user error and we should have continued to fail-fast, hopefully report the exception, and halt. Proceeding in such a case is not reasonable IMO but if you choose to do so then the prudent thing would be to ignore the errant thread completely - after reporting the IllegalThreadStateException. with hooks you have no idea even if you do start them but still ... :) [~smarks] This is all a can of worms when you start trying to discuss every little detail - good luck! See my comments on JDK-8290196 regarding documenting DestroyJavaVM.
14-07-2022
> What should happen if a shutdown hook is already running when the shutdown sequence begins? Or if in fact a shutdown hook has already run to completion? It appears that the Loom has made some behavior changes in this area. A shutdown hook is a Thread and it's always been possible to explicitly start it before exit - this would be user error of course but it has always been possible. The historical behavior was to abort the starting of any subsequent hooks and to completely skip the joining, meaning System.exit or the DestroyJavaVM thread may not start all hooks and would not wait any hooks to finish. The IllegalThreadStateException was caught and ignored. Project Loom had to look at this because a shutdown hook could be a virtual Thread and there are more things that could go wrong. The behavior with JDK 19 is IllegalThreadStateException is handled when attempting to start the thread/hook. If there are 5 shutdown hooks and 2 were previously started (due to user error) then it starts the remaining 3 and waits for all to finish. It could be argued that it should only join the hooks that were actually started during exit but that would require changes to existing spec. I agree this should be looked at, minimally more warning could be added to the spec to never start the threads specified to addShutdownHook.
14-07-2022
Some topics that need to be covered by the changes to the specifications here are as follows: * It would be good to have a more formal definition of _shutdown sequence_, probably in the Runtime class specification, instead of having it mixed into the method specification of addShutdownHook. This will make it easier to reference from other parts of the specs. * What should happen if a shutdown hook is already running when the shutdown sequence begins? Or if in fact a shutdown hook has already run to completion? It appears that the Loom has made some behavior changes in this area. Previously it seemed like the IllegalThreadStateException from starting such a shutdown hook wasn't handled, and it would stop the shutdown sequence immediately (not running any shutdown hooks that hadn't already been started, and not waiting for previously started ones to finish) and halt the VM immediately. Now such exceptions are ignored, and other shutdown hooks are started and will be allowed to complete before VM halt. (And hooks that were already started will be joined before VM halt.) Arguably this is better behavior, but there should be some discussion about what the behavior ought to be, and what ought to be specified. * A shutdown hook can be a daemon or non-daemon thread (despite the wording in the JLS 12.8 draft in JDK-8290196). However, this is mostly immaterial, as the hooks' daemon status aren't considered once the shutdown sequence has begun. However, a hook's daemon status IS significant if it's started explicitly before the shutdown sequence begins. In this case it counts as a regular non-daemon thread. * It should be clarified that there is very little special about VM's and the program's state during the shutdown sequence. The only special things are: 1) shutdown hooks can no longer be added or removed, and 2) there is a pending state such that, at the end of the shutdown sequence, the JVM is halted. Otherwise, daemon and non-daemon threads continue to run normally and can even create new daemon and non-daemon threads. There is a mention of "the last non-daemon thread" whose exit initiates the shutdown sequence. It's quite possible for there to be non-daemon threads running at the end of the shutdown sequence, so in no way is that one the "last" non-daemon thread. Better wording might be something like, "The shutdown sequence begins when the number of non-daemon threads reaches zero for the first time." * Handling of JNI_DestroyJavaVM should be clarified either here or in the JNI specification. I have a bunch of questions about this. It seems different from both System::exit and from the non-daemon thread count reaching zero. * The paragraph regarding "In rare circumstances the virtual machine may <i>abort</i>" should be cleaned up, since JLS 12.8 (JDK-8290196) seems to cover this case adequately. * There is also some possibly-redundant text in the spec of Runtime::exit that will need to be adjusted, in addition to the changes made by JDK-8288984. The phrase "never returns normally" is the same as "blocks indefinitely" which is also mentioned in JLS 12.8 (JDK-8290196). It should be made clear these are the same concept. It should immediately follow that if a shutdown hook calls Runtime::exit, since that method does not complete, the hook does not complete, and thus the shutdown sequence will not complete. This is a deadlock, but it isn't quite "the system will deadlock." Other threads (including shutdown hooks) continue to run just fine, and the JVM can be halted via a call to Runtime::halt. * System::exit needs some tweaks as well, probably something along the lines of "Initiates shutdown of the Java Virtual Machine."
13-07-2022
See draft PR here: https://github.com/openjdk/jdk/pull/9437
13-07-2022

CSR :	JDK-8294817 - Define and specify Runtime shutdown sequence
Relates :	JDK-8295857 - Clarify that resources within try-with-resources are not releasead after Runtime.halt()
Relates :	JDK-8290196 - 12.8: Clarify the definition of program exit
Relates :	JDK-8273302 - Clarify the specification of JNI DestroyJavaVM in relation to "unloading the VM"
Relates :	JDK-8290388 - 5.7: Clarify the definition of JVM termination
Relates :	JDK-4398224 - Introductory Thread class comments out of date with new shutdown protocol
Relates :	JDK-8290482 - Update JNI Specification of DestroyJavaVM for better alignment with JLS, JVMS, and Java SE API Specifications
Relates :	JDK-8288984 - Simplification in java.lang.Runtime::exit