Conflict when observability enabled and mixed batch and non-batch listeners

[michaldo]

In what version(s) of Spring for Apache Kafka are you seeing this issue?

3.3.6

Describe the bug

Batch @KafkaListener is not observable, non-batch is observable - it causes Prometheus metric conflict.

When observability is enabled and there is one non-batch @KafkaListener, metric
http://localhost:8080/actuator/prometheus?includedNames=spring_kafka_listener_seconds is ok:

spring_kafka_listener_seconds_sum{error="none",messaging_kafka_consumer_group="aaa",messaging_operation="receive",messaging_source_kind="topic",messaging_source_name="bbb",messaging_system="kafka",spring_kafka_listener_id="org.springframework.kafka.KafkaListenerEndpointContainer#0-0"} 0.0016962

When second, batch @KafkaListener is added, it breaks non-batch metrics

spring_kafka_listener_seconds_sum{exception="none",name="org.springframework.kafka.KafkaListenerEndpointContainer#0-0",result="success"} 0.0

Logs contains warning with explanation:

The meter (MeterId{name='spring.kafka.listener', tags=[tag(error=none),tag(messaging.kafka.consumer.group=aaa),tag(messaging.operation=receive),tag(messaging.source.kind=topic),tag(messaging.source.name=bbb),tag(messaging.system=kafka),tag(spring.kafka.listener.id=org.springframework.kafka.KafkaListenerEndpointContainer#1-0)]}) registration has failed

To Reproduce

1. Build application with

• org.springframework.kafka:spring-kafka
• org.springframework.boot:spring-boot-starter-actuator
• io.micrometer:micrometer-registry-prometheus
• org.springframework.boot:spring-boot-starter-web (for easy metric inspection)

Make sure: spring.kafka.listener.observation-enabled=true

Experiment with different combination of batch and non-batch @KafkaListener

Expected behavior

It cannot be accepted that adding batch listener breaks already working obsevations

When observability is enabled, metric for each listener should be registered with same set of tags, regardless listener is batch or non-batch). When some tag has no value, use placeholder "none"

Similar issue for MongoDB: spring-projects/spring-data-mongodb#4994

[sobychacko]

Somewhat related, we are reviewing this PR: #3944