RQ backend interfaces

[RealOrangeOne]

The RQ backend implementation is important, as it's the first backend which interfaces with a 3rd-party existing queue system.

The implementation works, but currently has a few places it could be better:

• Which worker ran a given execution ("result" in RQ) is currently smuggled through the job meta
• There doesn't seem to be an easy way to find a job based on its id regardless of which queue it's on. Instead, relying on looping through all queues to search for a job (more Redis API calls)
• In future, I'd like to distinguish between "the task raised an exception" and "something in the worker failed on this task". Currently, RQ doesn't seem to store the latter. For example, what does RQ do when the return type of a task isn't picklable?

Similarly, if experts in the internals of RQ / django-rq could review the RQ backend, I'd be very interested in ways we could either improve how we interface with RQ, or perhaps even update our API to better fit RQ or larger-scale workers.

[RealOrangeOne]

@selwin interested to hear your thoughts!

[selwin]

Which worker ran a given execution ("result" in RQ) is currently smuggled through the job meta

This is something I can help with. Always planned to add result.worker_name so we know which worker executed the job. I should be able to do this in one or two weeks.

There doesn't seem to be an easy way to find a job based on its id regardless of which queue it's on. Instead, relying on looping through all queues to search for a job (more Redis API calls)

You can do this:

job = Job.fetch('job_id', connection=redis)

In future, I'd like to distinguish between "the task raised an exception" and "something in the worker failed on this task". Currently, RQ doesn't seem to store the latter. For example, what does RQ do when the return type of a task isn't picklable?

So the question is to differentiate types of failure between job and RQ related failure. Will think about this.

Specific to the case when a job returns something unpickleable, this should still be marked as a successful execution, but result.value set to unpickleable return value or something like that - I'll have to check whether this is the current behavior.

[selwin]

Just a quick update

Which worker ran a given execution ("result" in RQ) is currently smuggled through the job meta

Done in rq/rq#2285 . Accessible via result.worker_name.

Specific to the case when a job returns something unpickleable, this should still be marked as a successful execution, but result.value set to unpickleable return value or something like that - I'll have to check whether this is the current behavior.

This is already handled in https://github.com/rq/rq/blob/93d70f75998bee4d6f7a37d572b7883786b3e90c/rq/results.py#L222 . Result will still be created properly, but result.return_value will return Unserializable return value.