One Consumer per Task or Shared Consumer? #14257
-
Community Support Policy
RabbitMQ version used4.1.x How is RabbitMQ deployed?Community Docker image Steps to reproduce the behavior in questionHi everyone, I’m using RabbitMQ streams, and I’m trying to figure out the best way to structure consumers for multiple tasks. Let’s say I have a stream with 5 partitions. If each task opens its own consumer, and I have 100 different tasks, that means 500 consumers (5 partitions × 100 tasks). If I reuse one consumer and handle all tasks inside, I only need 5 consumers total. My questions: Is it better to open one consumer per task, or handle multiple tasks inside a shared consumer? If I go with shared consumers, how do you typically handle acknowledging messages when some tasks succeed and others fail? Thanks |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
We have examples with thousands, and even tens of thousands of consumers. Tens of thousands of consumers will require more CPU resources and we would not recommend that many. As for shared consumers and streams, there cannot be one right solution because we do not know to what extent your consumers require coordination and can coordinate. Stream filtering is a relevant feature to mention here. |
Beta Was this translation helpful? Give feedback.
-
|
I dont know how stream filter helps here.
If each task is assigned a filter on the stream then again we need one
consumer per task.
We will save on resources since messages will not have to be delivered on
the consumers but again the consumer will have to open
…On Mon, 21 Jul 2025, 20:06 Michael Klishin, ***@***.***> wrote:
We have examples with thousands, and even tens of thousands of consumers.
Tens of thousands of consumers will require more CPU resources and we would
not recommend that many.
As for shared consumers and streams, there cannot be one right solution
because we do not know to what extent your consumers require coordination
and can coordinate. Stream filtering
<https://www.rabbitmq.com/blog/2023/10/16/stream-filtering> is a relevant
feature to mention here.
—
Reply to this email directly, view it on GitHub
<#14257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAV3YNYEQ46ZNNNQRZZBET3JUM2HAVCNFSM6AAAAACCAMYUYCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGOBTGYYTSNI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
Stream filters help with the number of streams, not consumers, that's correct. If a small number of consumers is enough for your needs, I'd start with a small number. |
Beta Was this translation helpful? Give feedback.
We have examples with thousands, and even tens of thousands of consumers. Tens of thousands of consumers will require more CPU resources and we would not recommend that many.
As for shared consumers and streams, there cannot be one right solution because we do not know to what extent your consumers require coordination and can coordinate. Stream filtering is a relevant feature to mention here.