Add support for bulk cancellations/kills/pause/unpause

[tillrohrmann]

In certain scenarios where one has accumulated many invocations that cannot complete, it would be very helpful to have the functionality to bulk cancel/kill those invocations. The criterion could be the creation timestamp of the invocation: Cancel/kill all invocations older than 30 minutes, for example. Ideally, this operation does not need to scan through all available invocations because if the cluster is already stressed, these queries would add even more stress.

[slinkydeveloper]

Other criterias we heard from users:

• by service
• by service handler
• by object key
• by deployment id -> this is asked to decommission an endpoint (in the context of Service deployment / registration issues #3144

Some of those require broadcast to the whole cluster obviously

[slinkydeveloper]

To retain a good amount of flexibility, also when we integrate this with the UI (something to be understood while we design this feature), it would be awesome to design in a more or less generic way the "bulk" operations:

• Given a filter (maybe even in SQL!)
• Given the operation I want to execute
• Execute it

Beware that not all operations are wal commands, some require to run an RPC Handler first (see Restart as new and pause and resume) or are entirely handled by an RPC Handler (don't have the example in the code yet, but Pause will be like that)

[tillrohrmann]

Bulk unpausing could also be a really helpful operation.

[slinkydeveloper]

I would like to start this from the UI, and then infer from there the implementation details in the admin api. I opened an issue there.

[nikrooz]

We can certainly start with the UI, but it’s important to note that running an action against a query is quite different from running an action against multiple selected invocations (e.g., from tables). In each case, the target of the action should be absolutely clear.

We can begin with the multiple-invocations version, but that doesn’t necessarily mean the same UI or implementation can be reused for query-based actions

[slinkydeveloper]

Answered on the other thread.

[slinkydeveloper]

#3943 makes much more usable the CLI command to kill/cancel/pause/etc in batch, I can kill 2 milion invocations easily: takes roughly a minute, but I mean it's a long operation and gives you back the feedback "was killed or not", so i would say it's an ok tradeoff for now.
There is an open issue for the UI side of things, to have a similar approach to the CLI one.

I would say let's finish those efforts and put them out, right now I have the feeling those might be "good enough", without the need for something more sophisticated on the Admin API with filters and stuff...