Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion Deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -549,7 +549,48 @@ When deploying Karafka consumers using Kubernetes, it's generally not recommende

For larger deployments with many consumer processes, it's especially important to be mindful of the rebalancing issue.

Overall, when deploying Karafka consumers using Kubernetes, it's important to consider the deployment strategy carefully and to choose a strategy that will minimize the risk of rebalancing issues. By using the `Recreate` strategy and configuring Karafka static group memberships and `cooperative.sticky` rebalance strategy settings, you can ensure that your Karafka application stays reliable and performant, even during large-scale deployments.
Overall, when deploying Karafka consumers using Kubernetes, it's important to consider the deployment strategy carefully and to choose a strategy that will minimize the risk of rebalancing issues. By using the `Recreate` strategy and configuring Karafka with appropriate rebalancing strategies, you can ensure that your Karafka application stays reliable and performant.

### Choosing the Right Rebalance Strategy

**For teams running Kafka 4.0+:**

- Use the new **KIP-848 consumer protocol** (`group.protocol` set to `consumer`) as your primary choice
- Benefits: Faster rebalancing, less disruption, simpler operation, improved static membership handling
- This is the recommended approach for all new deployments with compatible infrastructure

**For teams on older Kafka versions or with compatibility constraints:**

- Use **`cooperative-sticky`** rebalance strategy
- Still provides incremental rebalancing benefits over the older eager protocols
- Migrate to KIP-848 when your infrastructure supports it

Example configuration for KIP-848:

```ruby
class KarafkaApp < Karafka::App
setup do |config|
config.kafka = {
'bootstrap.servers': '127.0.0.1:9092',
# Use the new consumer protocol (KIP-848)
'group.protocol': 'consumer'
}
end
end
```

Example configuration for cooperative-sticky (fallback option):

```ruby
class KarafkaApp < Karafka::App
setup do |config|
config.kafka = {
'bootstrap.servers': '127.0.0.1:9092',
'partition.assignment.strategy': 'cooperative-sticky'
}
end
end
```

### Liveness

Expand Down
24 changes: 23 additions & 1 deletion Development-vs-Production.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,35 @@ Topics that are automatically created because of `allow.auto.create.topics` are

## Consider the Impact of Rolling Deployments on Rebalances

Whenever you do a rolling deployment of `N` processes, expect `N` rebalances to occur. Rebalances can impact the performance and stability of your Kafka cluster. However, using the `cooperative-sticky` rebalance strategy can mitigate some of these issues.
Whenever you do a rolling deployment of `N` processes, expect `N` rebalances to occur. Rebalances can impact the performance and stability of your Kafka cluster. Choosing the right rebalancing strategy can significantly mitigate these issues.

### Recommended Rebalancing Strategies

**For Kafka 4.0+:**

```ruby
class KarafkaApp < Karafka::App
setup do |config|
config.kafka = {
'bootstrap.servers': '127.0.0.1:9092',
# Use the new consumer protocol (KIP-848) - Recommended
'group.protocol': 'consumer',
'group.remote.assignor': 'uniform' # or 'range'
}
end
end
```

Benefits: Faster rebalancing, minimal disruption, improved static membership

**For older Kafka versions:**

```ruby
class KarafkaApp < Karafka::App
setup do |config|
config.kafka = {
'bootstrap.servers': '127.0.0.1:9092',
# Fallback to cooperative-sticky for older brokers
'partition.assignment.strategy': 'cooperative-sticky'
}
end
Expand Down
Loading