Lettuce failed to reconnect when connection is broken

[exceptionplayer]

I am using lettuce 6.5.2.RELEASE to connect to my redis cluster.

When something is wrong with the network, the connection will be broken, but it fails to reconnect when the network is normal.

From the logs I can see that, the connection is broken:

io.lettuce.core.RedisException: Currently not connected. Commands are rejected.
	at io.lettuce.core.protocol.DefaultEndpoint.validateWrite(DefaultEndpoint.java:312) ~[lettuce-core-6.5.2.RELEASE.jar!/:6.5.2.RELEASE/5271288]
	at io.lettuce.core.protocol.DefaultEndpoint.write(DefaultEndpoint.java:184) ~[lettuce-core-6.5.2.RELEASE.jar!/:6.5.2.RELEASE/5271288]
	at io.lettuce.core.protocol.CommandExpiryWriter.write(CommandExpiryWriter.java:125) ~[lettuce-core-6.5.2.RELEASE.jar!/:6.5.2.RELEASE/5271288]
	at io.lettuce.core.cluster.ClusterDistributionChannelWriter.writeCommand(ClusterDistributionChannelWriter.java:250) ~[lettuce-core-6.5.2.RELEASE.jar!/:6.5.2.RELEASE/5271288]
	at io.lettuce.core.cluster.ClusterDistributionChannelWriter.writeCommand(ClusterDistributionChannelWriter.java:236) ~[lettuce-core-6.5.2.RELEASE.jar!/:6.5.2.RELEASE/5271288]
	at io.lettuce.core.cluster.ClusterDistributionChannelWriter.doWrite(ClusterDistributionChannelWriter.java:185) ~[lettuce-core-6.5.2.RELEASE.jar!/:6.5.2.RELEASE/5271288]
	at io.lettuce.core.cluster.ClusterDistributionChannelWriter.write(ClusterDistributionChannelWriter.java:116) ~[lettuce-core-6.5.2.RELEASE.jar!/:6.5.2.RELEASE/5271288]

But I don't see any ReconnectAttemptEvent being logged, but the code in ConnectionWatchDog shows that it will publish a ReconnectAttemptEvent to eventbus each time a reconnect attempt is made.

Reconnect is import, but it does not seem to work properly, but it is not easy to reproduce this issue.

I am just here to seek some advice, anything will be helpful, thanks

[tishun]

Hey @exceptionplayer ,

• have you configured autoReconnect in a proper way? By default it is on, but you've not mentioned any sort of configuration, so this is a good place to start.
• ReconnectAttemptEvent is not logged by default, but a message saying "Reconnecting, last destination was {}" should be logged, if you allow INFO level logs; have you tried to listen programmatically for events using the eventBus? You can do it similar to how latency is being monitored here, but listening for ReconnectAttemptEvent
• reconnect has been working properly since it was introduced many versions before 6.5.2 and is actively tested by integration tests, so you are either hitting a very isolated bug, or have some configuration error

[exceptionplayer]

Hey @tishun, thank you for reply.

• have you configured autoReconnect in a proper way?

  • yes, because I have used the same code to test in our test env, it worked well. The failure to reconnect issue happened in our product environment, some machines can reconnect successfully within a few minutes(which is also strange) but some machines can't.

• ReconnectAttemptEvent is not logged by default.

  • yes, we have logged the event using the eventbus from ClientResources, also I checked the "Reconnecting, last destination was {}" log, nothing was logged when it failed to reconnect.

• so you are either hitting a very isolated bug, or have some configuration error

  • yes, it works well some times, it is not easy to reproduce.

There is one thing:
I was sending data(shared pub/sub) to 4 different redis clusters, I created 4 ClusterRedisClients(one for each redis cluster) and used each ClusterRedisClient to create a StatefulRedisClusterConnection, but the 4 ClusterRedisClients shared the same ClientResources, is it possible that some EventLoopGroup is busy?