Busy-waiting watchdog

[belm0]

It's easy to cause unwanted busy-waiting in tasks, either by mistake (e.g. left out a sleep() call within a periodic loop) or by poor design (e.g. relying too heavily on polling rather than events). The simplest valid Trio program which will use all headroom on the CPU is while True: trio.sleep(0).

This is potentially far more common than the "task never awaits" scenario being addressed in #596.

I'd like to find a heuristic that can detect such problematic code while avoiding false positives.

some metrics which could be employed: median task block time vs. task CPU time over last T seconds, task schedule rate, ranking task schedule rate, cumulative task CPU time over the last T seconds, cumulative CPU time of Trio scheduler over the last T seconds

[belm0]

based on scheduler "duty cycle":

duty cycle of the trio scheduler is a percentage defined as scheduling_wall_time / total_time for a given period of interest, like 5 seconds. scheduling_wall_time is time spent in the task scheduler loop other than 1) blocking on tasks to schedule and 2) executing tasks. High duty cycle implies a busy-wait. In that case, perhaps we can assume the cause is the one or more top tasks in terms of scheduling rate. Since tasks sharing the same bad implementation can collude, we probably want to report the top task and anything within some sigma from it.

Haven't looked at the trio scheduler, so no idea if these indicators are feasible.