Naive back-off delay retry solution

[dabeeeenster]

First - thanks for this package - Django has been needing it for too long!

We're currently using huey and do like it, but I would rather drop redis and use our core DB for the task management. We hit a lot of external APIs in our platform and often have to deal with flakiness, so retry and back-off delays are important to us. I had a play around with a super basic retry solution (with back-off delay!) that looks like this:

@task()
def calculate_meaning_of_life(retry_number: int) -> bool:
    if retry_number > 10:
        raise Exception(
            f"Failed to calculate the meaning of life, retry_number={retry_number}"
        )
    else:
        # Do some work here
        ...

        # Simulate a failure, so execute a recursive retry
        calculate_meaning_of_life.using(
            run_after=timezone.now() + timedelta(seconds=pow(2, retry_number))
        ).enqueue(retry_number + 1)

I thought it might be useful for others, or if people have suggestions for improvements?

In terms of the retry logic/engine for this package, has any decision been made on whether this package itself will be responsible for that? Will it delegate to the specific backend? Or a choice?

[RealOrangeOne]

I think adding too much complexity to how the backoff is calculated is probably doomed to scope creep, but a statically defined backoff as part of the retry could definitely be useful.

See also #6 for retrying in general.