MQTT QoS 1 failure loop when 7k device connect

[minhnguyenvan95]

Describe the bug

Incidence happen in 9:40

More than 7k4 devices connected. Then suddenly the queues dropping to 2k8. The queue keeping dropping.

Firewall connection limit by 8k and connection allow 24 connection/s

Rabbitmq overview

Log in rabbitmq

2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0> Failed to declare queue 'mqtt-subscription-REDACTED_USERNAMEqos1' in vhost '/': {existing, 2025-07-28T09:59:59.700670796+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                               {amqqueue, 2025-07-28T09:59:59.700683783+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                {resource, 2025-07-28T09:59:59.700695251+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                 <<"/">>, 2025-07-28T09:59:59.700746249+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                 queue, 2025-07-28T09:59:59.700758834+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                 <<"mqtt-subscription-REDACTED_USERNAMEqos1">>},
2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                true, 2025-07-28T09:59:59.700822815+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                false, 2025-07-28T09:59:59.700841664+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                <25558.17522573.0>, 2025-07-28T09:59:59.700856779+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                [], 2025-07-28T09:59:59.700868517+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                <25558.17490532.0>,
2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                [], 2025-07-28T09:59:59.700903873+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                [], 2025-07-28T09:59:59.701001834+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                [],
2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                undefined, 2025-07-28T09:59:59.701015988+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                undefined, 2025-07-28T09:59:59.701021740+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                [], 2025-07-28T09:59:59.701027091+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                [], 2025-07-28T09:59:59.701032595+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                live, 2025-07-28T09:59:59.701038048+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                0,
2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                [], 2025-07-28T09:59:59.701050148+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                <<"/">>,
2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                #{user => 2025-07-28T09:59:59.701063807+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                   <<"REDACTED_USERNAME">>}, 2025-07-28T09:59:59.701069601+07:00 2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                                                rabbit_classic_queue,
2025-07-28 09:59:59.699184+00:00 [error] <0.17495494.0>                                                       

Reproduction steps

1. RabbitMq Mqtt run in k8s with 10 pod ( installed by helm )
2. MetalLb port 1883 before 10 Mqtt -> port 1883
3. Firewall before MetalLb ( Firewall handle TLS ) 8883 -> clear text MetalLb 1883
4. RabbitMq enable auth plugin ( http auth backend controll for the MQTT access for iot devices )
5. More than 7100 iot devices connected, then sometime all server go down with the loop failure
6. The iot device ping interval 1000ms to server, if the ping failure the iot will reconnect to the sever ( LOOP )
7. The Rabbitmq overview show the connection increase but the queue is decrease ( sometime connection become 28k )

Expected behavior

The mqtt protocol still work with more than 25k connected devices with qos 1

Additional context

No response

[minhnguyenvan95]

The auth backend seem not an issue, because the log in auth backend http show the response so fast. The issue could be internal mqtt and rabbitmq system

[minhnguyenvan95]

10:20 We have to disable all the subscriber ( limit Firewall and load balancer ) then increase the connection limit by step 500 ( limit concurrence connection to 10/s ).

After this the connection and queues keep increase after this then the system working normally

[minhnguyenvan95]

Clean session from client was false

[michaelklishin]

The issue could be internal mqtt and rabbitmq system

You haven't provided any evidence of that. No logs, not even what RabbitMQ version is used.

After this the connection and queues keep increase after this then the system working normally

RabbitMQ does not those open connections, your application does. Connection churn and leaks are so common, there is a dedicated doc section on both topics.