Negative consumer availablePermits

[backuitist]

Under which circumstances a consumer can end up with a negative availablePermits?

I'm on Pulsar 4.0.4 and here is a snippet of the topics stats:

          "subscriptions": {
            "<sub-123>": {
              "consumers": [
                {
                  "address": "/10.0.5.78:38698",
                  "consumerName": "qQ9mm",
                  "availablePermits": -1727,

[lhotari]

Yes, I've seen this before. Some analysis at #23263 .

I guess it might be related to batch messages and the mitigation could be to use preciseDispatcherFlowControl=true in broker.conf to more accurately dispatch messages to consumers according to the available permits. The other option is that there's a bug somewhere when permits handled. If the client sends permits for batches, but the broker expects permits at message level (for each individual message in the batch), that would obviously be a problem.

[backuitist]

Thanks @lhotari , I was also suspecting batching to be involved as we produce batches potentially larger than the consumer receiver queue size.

[FlorentinDUBOIS]

We had an issue with the batching handling in pulsar-rs which does not count the same way as the broker and got negative available permits.

We fixed the issue in the following commit streamnative/pulsar-rs@ba58c9c. But, if I am here is that I got the same issue in production again 😢.