Consider dropping `incremental_mutants` CI job

[tnull]

It seems to me that the incremental_mutants CI job is failing on ~every PR, and nobody actually makes any effort to address the failures. IMO it therefore just introduces noise which then in turn might have us getting used to a failing CI before merging, i.e., has some potential that an actual issue slips through at some point. Of course, it also takes up some (considerable?) time in the CI queue that might be better spent on other jobs.

I'd therefore like to ask whether there are any contributors that actually look at the CI job. Does somebody find it useful and does spend some time to make it pass on any of their PRs?

Otherwise, if nobody actually gets any use out of it, I'd be in favor of disabling it again.

(cc @dunxen)

[dunxen]

Yeah, you are right. It's still at a point where it's not useful, since so many mutations let tests pass across the codebase. Even with some of the smallest diffs, there are often pre-existing failures.

We could disable CI and have some easy-to-run script in the ci dir for running locally if desired. So that we could eventually work towards "fixing" the failures.

[martinsaposnic]

I haven't dug deeper yet, but I've noticed a few cases where a reported MISSED turned out to be a false positive. When I manually applied the suggested change, the tests failed, so it wasn't actually a MISSED.

Not sure if it's a bug in the tool, a config issue, or something else.

I don't have an example right now, but I'll share one next time it happens.

All that said, I do find it somewhat useful (I wouldn't mind running it locally), but it's frustrating when it doesn’t work as expected.

[TheBlueMatt]

It might also be something we consider looking at in the future on specific parts of the codebase (eg some of the channel state machine), but disabling for the rest of the codebase.

[tnull]

Now opened #3953 and #3954. Drafts for now, until we reached a conclusion.