Task deduplication

[oliverhaas]

One of the features I think the ideal task queue needs is task deduplication.

My use cases are usually that I either want to just "drop" the second duplicate task if the same task is already queued, or that I want to "replace" the first task with the second, so basically a debounce.

Same as for deferred/run_after tasks, I expect that some backends will not be able to do this, but for the default DB backend it should be fairly straight forward.

Does this make sense to others here?

[RealOrangeOne]

I'm interested in the use case for this. When would a task be duplicated? Are you imagining it be a bug, or just caused by an impatient user (ie they click "download report" twice, and thus get 2 emails).

I suspect the way to do this is to determine whether a similar task is in the queue, and just use that instead. You should be storing the task id somewhere outside of the queue anyway, so hopefully it should be reasonable simple to determine.

[oliverhaas]

I'm interested in the use case for this. When would a task be duplicated?

Duplicated tasks can happen just like you said; they just get triggered twice in one way or another.

One of my projects serves images and videos in different resolutions, and if a specific resolution does not exist, I'll return the closest available one, but also trigger a task to do the conversion for the exact requested resolution. User hits reload before the conversion finishes -> duplicate task.

The workaround (I see it as that) is of course to double check manually in the task or before triggering a task whether something is already queued up, or have some kind of bookkeeping, but I really think a task or message queue is much easier to use if it's fire-and-forget in a way.

I suspect the way to do this is to determine whether a similar task is in the queue, and just use that instead. You should be storing the task id somewhere outside of the queue anyway, so hopefully it should be reasonable simple to determine.

Yeah. It should be some kind of id (which could be a hash of your own choosing) which would determine which tasks are "the same". Google Tasks for example does it like this, see https://cloud.google.com/tasks/docs/reference/rest/v2beta3/projects.locations.queues.tasks/create#body.request_body.FIELDS.task.

---

Also I like to use task deduplication for debounces. I have a project where data is supposed to be synchronized to another system 5 minutes after the last change happened. I think just having a tasks queue which can deduplicate/debounce is easier than other solutions.

[RealOrangeOne]

I could see a value in some kind of idempotency token, which is used for the comparison. I'm not sure if it's valuable having it as a dedicated concept, or whether there's tooling to implement it yourself.

[Benoss]

I have a use case for sync to external system.
We have a queue to update an external system, this happens when a user logs in, or when some objects are modified in DB for example.
The external system is not very scalable (outside of my control) and super slow with aggressive rate limit.

If there is an idempotency token and the task with this token is not currently being processed I want to insert a new task. On the other hand if a task with this token already exists I do not need to insert the task as I know that it will be synced already.
Because the target system is slow this would ensure that when the queue is big we do not duplicate tasks that are going to republish the same thing to the external system.

[RealOrangeOne]

It's a real hack, and you probably shouldn't do it, but you could achieve this by querying the DBTaskResult table directly for tasks with the matching args/kwargs pattern.

In reality, I suspect that kind of idempotency is better handled by a model in your business logic, rather than a function of the task queue itself.

[Benoss]

Let's say that I don't want to re-insert a task with those kwargs if it is already in the queue but not currently being processed
{'obj': 'company', 'id': 213456}
How would you approached this, if the task with those kwargs is currently being processed.
Doing a json query on the nested values of kwargs seem inefficient on the other hand I could generate my own key
sha1(obj + id) for example and then it is my responsibility to make it unique within my app.

I suppose the only thing I am asking is for an easy way to skip enqueue if a string already exists