First consumer takes all messages - how to share the load when scaling up?

[idohuber]

Use Case Overview:

We have a specific use case that we're struggling to configure properly in Pulsar:

• Message Pattern: Small bursts of messages (10-100) arrive at unpredictable times, queue is empty most of the day
• Processing Time: Each message requires 15-80 minutes of processing (CPU-intensive tasks)
• Scaling Requirement: We have a controller that monitors the backlog and automatically scales consumers up/down based on demand
• Problem: When messages arrive before consumers are scaled up, the first consumer gets ALL messages, and later-added consumers remain idle

Our Setup:
consumer, err := c.client.Subscribe(pulsar.ConsumerOptions{
Topic: c.topic,
SubscriptionName: c.subscription,
Type: pulsar.Shared,
Name: consumerID,
ReceiverQueueSize: 1,
}
each msg is being sent with different key

The Core Problem We Discovered:
Through extensive testing, we found that when messages exist in the backlog before consumers join:

1. First consumer to connect gets ALL messages assigned by the broker
2. When additional consumers are added by our auto-scaler, they get 0 messages
3. New consumers only receive newly-arriving messages, not the existing backlog
4. This leads to extremely uneven load distribution where 1 consumer processes for hours while 9 sit idle

The Configuration We're Looking For:
Is there a broker-side configuration that would:

• Limit the maximum number of messages assigned to each consumer at the broker level (not just client-side)
• Allow messages to be redistributed when new consumers join
• Something like maxMessagesPerConsumerBrokerSide or maxPendingMessagesPerConsumer

Essentially, we need the broker to only assign 1 messages to each consumer each time, keeping the rest in the backlog for other consumers to claim.

Environment

• Pulsar Version: 3.0.3
• Deployment: k8s jobs as consumers, each job is a consumer
• Client: github.com/apache/pulsar-client-go v0.12.1

Any guidance would be greatly appreciated. We really like Pulsar for our other use cases but struggling with this specific scenario.

thanks ahead!