Implement CI scheduler #5
Description
As suggested by Mark Wielaard in the mailing list:
But this is a real concern and will happen if you are too eager testing
all the things all the time. So I would recommend to start as small as
possible. Pick a target that builds as fast as possible. Once you go
over 30 minutes of build/test time it really is already too long. Both
on machine resources and human attention span. And pick a subset of the
testsuite that should be zero-FAIL. Only then will people really take
notice when the build turns from green-to-red. Otherwise people will
always have an excuse "well, those tests aren't really reliable, it
could be something else". And then only once you have a stable
small/fast builder that reliably warns committers that their commit
broke something extend it to new targets/setups/tests as long as you
can keep the false warnings as close to zero as possible.
In short to implement a CI scheduler which is not longer than a coffee break:
- choose a fast configuration to compile;
- choose a subset of the tests that take less than 30mins;
- ensure the subset is FAIL free and stays fail free;
if there are failures we stop and notify people. If there are no fails we can possibly trigger a full test.