failed to update dispatcher pods annotation event from broker-controller in the Broker objects

[pablo-suazo]

Describe the bug
After upgrading from knative 1.16 to 1.17 we are starting to see some events in the brokers: failed to update dispatcher pods annotation
This is happening as well after upgrading to knative 1.18

Expected behavior
The Broker has no Warnings in the Events section

To Reproduce
We are running it in an EKS cluster 1.32.7
Create the following Kafka Broker

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
  annotations:
    # This makes sure we create a Knative Kafka Broker (there are different kinds of Brokers)
    eventing.knative.dev/broker.class: Kafka
  labels:
    cg_subsystem: saas_infinity_platform
    cg_system: saas_infinity_platform
    cg_team: saas_infrastructure
  name: knative-test-pablo
  namespace: test-pablo
spec:
  config:
    apiVersion: v1
    kind: ConfigMap
    name: kafka-broker-config
    namespace: knative-eventing

Wait until the broker is ready.
Restart the kafka-controller pod.
Every time the kafka-controller is restarted or after some time, a new event is generated

kubectl describe broker -n test-pablo knative-test-pablo

Name:         knative-test-pablo
Namespace:    test-pablo
Labels:       cg_subsystem=saas_infinity_platform
              cg_system=saas_infinity_platform
              cg_team=saas_infrastructure
Annotations:  eventing.knative.dev/broker.class: Kafka
              eventing.knative.dev/creator: admin
              eventing.knative.dev/lastModifier: admin
API Version:  eventing.knative.dev/v1
Kind:         Broker
Metadata:
  Creation Timestamp:  2025-09-02T14:47:19Z
  Finalizers:
    brokers.eventing.knative.dev
  Generation:        1
  Resource Version:  1853066229
  UID:               13b1a491-e748-4800-8b42-be1a5fff42bd
Spec:
  Config:
    API Version:  v1
    Kind:         ConfigMap
    Name:         kafka-broker-config
    Namespace:    knative-eventing
  Delivery:
    Backoff Delay:   PT0.2S
    Backoff Policy:  exponential
    Retry:           10
Status:
  Address:
    Name:  http
    URL:   http://kafka-broker-ingress.knative-eventing.svc.cluster.local/test-pablo/knative-test-pablo
  Addresses:
    Name:  http
    URL:   http://kafka-broker-ingress.knative-eventing.svc.cluster.local/test-pablo/knative-test-pablo
  Annotations:
    auth.secret.ref.name:              kafka-tls
    bootstrap.servers:                 apac-vegemite-infinity-stg.a.aivencloud.com:13356
    default.topic:                     knative-broker-test-pablo-knative-test-pablo
    default.topic.partitions:          12
    default.topic.replication.factor:  3
  Conditions:
    Last Transition Time:  2025-09-02T14:47:21Z
    Status:                True
    Type:                  Addressable
    Last Transition Time:  2025-09-02T14:47:20Z
    Reason:                Config map knative-eventing/kafka-broker-brokers-triggers updated
    Status:                True
    Type:                  ConfigMapUpdated
    Last Transition Time:  2025-09-02T14:47:20Z
    Status:                True
    Type:                  ConfigParsed
    Last Transition Time:  2025-09-02T14:47:20Z
    Status:                True
    Type:                  DataPlaneAvailable
    Last Transition Time:  2025-09-02T14:47:20Z
    Message:               Feature "authentication-oidc" must be enabled to support Authorization
    Reason:                OIDCDisabled
    Status:                True
    Type:                  EventPoliciesReady
    Last Transition Time:  2025-09-02T15:27:33Z
    Status:                True
    Type:                  ProbeSucceeded
    Last Transition Time:  2025-09-02T15:27:33Z
    Status:                True
    Type:                  Ready
    Last Transition Time:  2025-09-02T14:47:20Z
    Reason:                Topic knative-broker-test-pablo-knative-test-pablo created
    Status:                True
    Type:                  TopicReady
  Observed Generation:     1
Events:
  Type     Reason                                       Age   From               Message
  ----     ------                                       ----  ----               -------
  Warning  failed to update dispatcher pods annotation  60m   broker-controller  Operation cannot be fulfilled on pods "kafka-broker-dispatcher-3": the object has been modified; please apply your changes to the latest version and try again
  Warning  failed to update dispatcher pods annotation  60m   broker-controller  Operation cannot be fulfilled on pods "kafka-broker-dispatcher-4": the object has been modified; please apply your changes to the latest version and try again
  Warning  failed to update dispatcher pods annotation  60m   broker-controller  Operation cannot be fulfilled on pods "kafka-broker-dispatcher-2": the object has been modified; please apply your changes to the latest version and try again
  Warning  failed to update dispatcher pods annotation  23m   broker-controller  Operation cannot be fulfilled on pods "kafka-broker-dispatcher-0": the object has been modified; please apply your changes to the latest version and try again
  Warning  failed to update dispatcher pods annotation  23m   broker-controller  Operation cannot be fulfilled on pods "kafka-broker-dispatcher-4": the object has been modified; please apply your changes to the latest version and try again

Complete logs: kafka-controller.log.zip
These are the kafka controller log entries related with this Broker

{"level":"info","ts":"2025-09-02T15:27:33.047Z","logger":"kafka-broker-controller","caller":"controller/controller.go:548","msg":"Reconcile succeeded","commit":"a5c7f94-dirty","knative.dev/pod":"kafka-controller-748665966c-fcllt","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.broker.Reconciler","knative.dev/kind":"eventing.knative.dev.Broker","knative.dev/traceid":"74bbf0a7-a55e-4e37-8e31-7c2d17750b47","knative.dev/key":"test-pablo/knative-test-pablo","duration":0.32322858}
{"level":"warn","ts":"2025-09-02T15:27:36.295Z","logger":"kafka-broker-controller","caller":"broker/broker.go:246","msg":"Failed to update dispatcher pod annotation to trigger an immediate config map refresh","commit":"a5c7f94-dirty","knative.dev/pod":"kafka-controller-748665966c-fcllt","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.broker.Reconciler","knative.dev/kind":"eventing.knative.dev.Broker","knative.dev/traceid":"dce173d6-b2d6-4255-99aa-a6ef912226fe","knative.dev/key":"test-pablo/knative-test-pablo","action":"reconcile","uuid":"13b1a491-e748-4800-8b42-be1a5fff42bd","error":"Operation cannot be fulfilled on pods \"kafka-broker-dispatcher-4\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"info","ts":"2025-09-02T15:27:36.295Z","logger":"kafka-broker-controller","caller":"controller/controller.go:548","msg":"Reconcile succeeded","commit":"a5c7f94-dirty","knative.dev/pod":"kafka-controller-748665966c-fcllt","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.broker.Reconciler","knative.dev/kind":"eventing.knative.dev.Broker","knative.dev/traceid":"dce173d6-b2d6-4255-99aa-a6ef912226fe","knative.dev/key":"test-pablo/knative-test-pablo","duration":0.317114719}
{"level":"info","ts":"2025-09-02T15:27:36.295Z","logger":"kafka-broker-controller.event-broadcaster","caller":"record/event.go:377","msg":"Event(v1.ObjectReference{Kind:\"Broker\", Namespace:\"test-pablo\", Name:\"knative-test-pablo\", UID:\"13b1a491-e748-4800-8b42-be1a5fff42bd\", APIVersion:\"eventing.knative.dev/v1\", ResourceVersion:\"1853066229\", FieldPath:\"\"}): type: 'Warning' reason: 'failed to update dispatcher pods annotation' Operation cannot be fulfilled on pods \"kafka-broker-dispatcher-4\": the object has been modified; please apply your changes to the latest version and try again","commit":"a5c7f94-dirty","knative.dev/pod":"kafka-controller-748665966c-fcllt"}

Knative release version
We are using the knative-operator 1.18.1 to deploy both knative-serving 1.18.0 & knative-eventing 1.18.1
We are using deploying eventing-kafka by using the manifests:

• https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.18.0/eventing-kafka-channel.yaml
• https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.18.0/eventing-kafka-broker.yaml
• https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.18.0/eventing-kafka-sink.yaml

[matzew]

This goes away, when using the 1.16 based broker?

[pablo-suazo]

This goes away, when using the 1.16 based broker?

This started to happen when we upgraded from the 1.16 based broker to the 1.17. It still happens with the 1.18 based broker

[matzew]

Does it go away when going back to 1.16?
Was there a change regarding your kafka?

Or was the only change the eventing version update?

[pablo-suazo]

We are not rolling back to 1.16, so I have no answer for that.
The Kafka cluster we are using did not have any version upgrade, we are using 3.8
We upgraded only knative eventing + eventing kafka broker

[matzew]

I checked, and we have seen these too. Those Operation cannot be fulfilled on ...: the object has been modified; please apply your changes to the latest version and try again errors are "normal" with knative controllers ... so it's really just a matter of not exposing these as "k8s events".

I guess there is no actual defect here, right?

[pablo-suazo]

What you mean with this? just a matter of not exposing these as "k8s events"
Do you have to make a change in the knative controller to not publish those events or we should make a change to filter these events?

[matzew]

nah, its something on our end

[pablo-suazo]

Are you releasing a patched version where these events are not exposed?

[twoGiants]

Hi @pablo-suazo! I am taking care of it.

/assign

[twoGiants]

@pablo-suazo issue will be fixed after this PR is merged #4548.

[pablo-suazo]

Thank you @twoGiants ! Looking forward to have it merged