Timeout handlers do not execute when the corresponding `abort on X timeout = True`

[MetRonnie]

[scheduler]
    [[events]]
        abort on workflow timeout = True
        workflow timeout = PT1S
        abort handlers = echo "ALPHA"
        workflow timeout handlers = echo "TANGO"

The workflow timeout handler is not running when abort on workflow timeout is set

Originally posted by @MetRonnie in #5959 (comment)

[oliver-sanders]

The reason for this is that when the workflow aborts, it terminates processes in the subprocpool INCLUDING the event handler.

Changing this behaviour will require careful thought as it could trigger events we don't want. E.G. preparing tasks may go into the submit-failed state erroneously.

[hjoliver]

Can we just leverage the cylc stop --now (but not --now --now) code, for the abort shutdown?

stop  -n, --now             Shut down without waiting for active tasks to
                        complete. If this option is specified once, wait for
                        task event handler, job poll/kill to complete. If this
                        option is specified more than once, tell the workflow
                        to terminate immediately.

[oliver-sanders]

Abort events take down the scheduler by raising a SchedulerError rather than requesting a shutdown. It's a much more instantaneous stop which also results in a non-zero exit code:

cylc-flow/cylc/flow/scheduler.py

Lines 1662 to 1663 in 9d985f2

	# "cylc play" needs to exit with error status here.
	raise SchedulerError(f'"{abort_conf}" is set')

In the abort case, we want to wait for aborted/timeout handlers, but I guess we might not want to wait for log file retrieval, etc (it could be a really critical shutdown).

[oliver-sanders]

A simple solution to this might be to run these timeout handlers behind "nohup" and skip the subprocpoll's "kill" command for them.

$ cat mess.py 
from subprocess import Popen

proc = Popen(['nohup', 'sleep', '100'])

$ python mess.py 
nohup: ignoring input and appending output to 'nohup.out'

$ ps -fu <me> | grep sleep | grep -v grep --color=never
<me> 1195133    5887  0 13:36 pts/3    00:00:00 sleep 100

Note, redirect to /dev/null to avoid creating a nohub.out file.

[oliver-sanders]

(This has been raised as a mid-high priority, pushing onto the 8.7.0 milestone)