Request Hedging, slow start

[ltagliamonte-dd]

Hello Folks,
I'm using lettuce client with apache kvrocks (redis on rockdb).
It is performing pretty good for my use case but when a node bootstraps from backup all its caches are cool and I see a small error rate.
I've added an init script that is warming up the shard and it has helped a lot, however it is not perfect.

Based on the fact the also valkey is looking into adding data tiering on disk I think it could be beneficial for everyone look into adding to the redis clients features like node slow start or requests hedging when the new redis node reports an uptime of N seconds (configurable)

My knowledge of this codebase is very low, but If someone could guide me i'm happy to contribute!
Thanks!

[tishun]

Hey @ltagliamonte-dd ,

could you share with us the mechanism you are using yourself for that? You can omit any sensitive data, the team would benefit from the general idea.

TBH we've had such requests before, but it is not on our radar right now. As usual any contributions (of any kind) could reduce the time we would need to get it out.

[ltagliamonte-dd]

Hello @tishun,
thanks for getting back to me on this!

The current warmup mechanism I've implemented, before bootstrapping kvrocks opens all the SST files, with 2 iterators (from beginning and from bottom) samples N keys from the file, starts kvrocks and uses those keys to perform HGETALL (my dataset uses redis hash).

In this way the FS cache, and other internal rocksdb data structures get warm.
It is working great but not perfect when a new node/pods gets "online" (ready to receive user traffic) we still record a small error rate client side.

From what I see, when the client receives a new node from cluster nodes, the client updates its topology and executes INFO on the node.
INFO returns the uptime of the new node, so what I was thinking is that for nodes with an uptime of less then N seconds we could do requests hedging or slow start.

[tishun]

Apologies for having to ask some more, but wanted to be sure we are on the same page here.

When you say node slow start what do you think the driver should do exactly?

For the request hedging I assume you would imagine that the driver sends requests to the master and all replica nodes and return the first received value, correct?

[ltagliamonte-dd]

Hello @tishun,
for node slow start i'm envisioning something like:

• 1 shards with 3 replicas, one of the replica nodes is "new", when reading the driver select the new node less times than the old nodes (for a specified amount of time only).

for request hedging I'm thinking:

• 1 shard with 3 replicas, one of the replica nodes is "new", when reading the driver sends the request to the new node and to an old node (for a specified amount of time only), uses the first one that returns.

I believe hedging is much simpler to implement.

[tishun]

Hey again,

for node slow start i'm envisioning something like:

• 1 shards with 3 replicas, one of the replica nodes is "new", when reading the driver select the new node less times than the old nodes (for a specified amount of time only).

This could be simulated with a custom implementation of ReadFrom setting. If you implement the ReadFrom abstraction in away that preserves the list of known nodes you can then compare the list every time a new connection is being established. Nodes that are provided to the select(Nodes nodes) method, that were not part of the call the last time, are obviously new nodes. Once added as "new nodes" you can set a custom timeout until they become commonly available.

This does not require any changes to the driver currently.

for request hedging I'm thinking:

• 1 shard with 3 replicas, one of the replica nodes is "new", when reading the driver sends the request to the new node and to an old node (for a specified amount of time only), uses the first one that returns.

This would not be a small change to the driver internals, tbh. I do not think we can do it easily, because not all commands are GET commands - any blocking, transactional or writing command could completely destroy the consistency of the system. We would obviously have to limit this feature to only read commands and even then we would have to consider how this would affect the performance of the system - for example the fact that we cancel a future for a command does not mean we can just easily ignore the result when it eventually arrives over the wire, because the nature of the driver is multiplexing and we would disturb the order of processing the commands. So we will be building up the queue of commands with slow commands, even if they were never meant to be sent to this node specifically.

IMHO this approach is not suitable for this driver.

What do you think, can you try to use approach 1?