Multi-region support for federation broker collective

[Msz-cx]

Hi folks,

I'm new to Choria and working on designing a resilient deployment using federation. I’ve currently deployed a federation broker collective within a single region (us-west-2) and use it as middleware for connecting regional Choria brokers. This works well, but I realized this introduces a single-region failure risk.

To address this, I’m exploring a multi-region setup and had an idea that I’m hoping to get some feedback on:

Proposed Setup (Solution 1):

• Deploy two separate Choria federation clusters, one in each region (e.g., primary and secondary).
• Have each regional broker connect to both federation clusters simultaneously.
• Register all federation brokers (from both clusters) in DNS under _mcollective-federation_server._tcp.

My thought was that federation clusters don’t need to be aware of each other, so when i run mco ping, as long as regional brokers ldn1 send responses to all federation brokers, then from Fed A side, each broker in Fed A received a response from ldn1, so can reach to a consensus, and same story for Fed B

However, in practice, when I run mco ping from one of the regional brokers, the results are unstable:

• Sometimes I get all expected node replies,
• Sometimes only partial responses,
• Sometimes none at all.

I’ve checked logs and didn’t find any clear errors. My question is:

**Is this kind of multi-cluster federation setup fundamentally incompatible with how Choria federation routing and message delivery works? Or is there a way to make this configuration reliable? If it's incompatible, which part causes the issue? **

Alternatively, I’m considering a single federation cluster that spans multiple regions, but before moving forward, I’d love to confirm whether my original idea is theoretically flawed, or if I may have just misconfigured something.

Thanks in advance for any guidance!

[ripienaar]

The best is to deploy a regional cluster so each region is standalone and connects to a nearby broker cluster

you then make one or two federation clusters that connects to all the regions for central admin

this way each region is resilient and you can have resilient federation also - however you can also connect to individual regions so imo making redundant federation probably is overkill

but for sure each region needs its own cluster.

[vjanelle]

Also depends on if you need to always be able to address the remote site or not.

[Msz-cx]

Yeah, that's why I chose to work on building two federation clusters first.. but I don't know why it's not working... To give a better understanding of what I'm having:

## SRV records used by Fed-A:
_mcollective-federation_server._tcp IN  SRV 0 0 4222  choria1.fed-A.example.net.
                                    IN  SRV 0 0 4222  choria2.fed-A.example.net.
                                    IN  SRV 0 0 4222  choria3.fed-A.example.net.

_mcollective-server._tcp        IN  SRV 0 0 4222  choria1.fed-A.example.net.
                                IN  SRV 0 0 4222  choria2.fed-A.example.net.
                                IN  SRV 0 0 4222  choria3.fed-A.example.net.

## SRV records used by Fed-B:
_mcollective-federation_server._tcp  IN  SRV 0 0 4222  choria1.fed-B.example.net.
                                     IN  SRV 0 0 4222  choria2.fed-B.example.net.
                                     IN  SRV 0 0 4222  choria3.fed-B.example.net.

_mcollective-server._tcp         IN  SRV 0 0 4222  choria1.fed-B.example.net.
                                 IN  SRV 0 0 4222  choria2.fed-B.example.net.
                                 IN  SRV 0 0 4222  choria3.fed-B.example.net.

## SRV records used by LDN:
_mcollective-federation_server._tcp  IN  SRV 0 0 4222  choria1.fed-A.example.net.
                                     IN  SRV 0 0 4222  choria2.fed-A.example.net.
                                     IN  SRV 0 0 4222  choria3.fed-A.example.net.
                                     IN  SRV 0 0 4222  choria1.fed-B.example.net.
                                     IN  SRV 0 0 4222  choria2.fed-B.example.net.
                                     IN  SRV 0 0 4222  choria3.fed-B.example.net.

_mcollective-server._tcp            IN  SRV 0 0 4222  choria1.ldn.example.net.
                                    IN  SRV 0 0 4222  choria2.ldn.example.net.
                                    IN  SRV 0 0 4222  choria3.ldn.example.net.

If I change LDN's _mcollective-federation_server._tcp to only point to one federation cluster, then I'm able to receive all nodes response consistently. But if I point to both cluster, and when i run mco ping from one of the federation cluster, I'm seeing sth like this

$mco ping
choria1.ldn.example.net. time=33.04 ms

---ping statistics ---
1 replies 

$mco ping
---ping statistics ---
0 replies

$mco ping
choria1.ldn.example.net. time=33.04 ms
choria2.ldn.example.net. time=33.04 ms
choria3.ldn.example.net. time=33.04 ms
---ping statistics ---
3 replies

Any ideas why it's going wrong?

[vjanelle]

I don't have the tooling in front of me to check, but you can crank up the debugging and there are tracing tools so that yo can see the topics and subjects that are being created by NATS and to confirm connectivity.

However I just created a centralised cluster, I didn't try to do what you're trying to do.

[ripienaar]

You will not be able to point the same client to both fed clusters the replies goes back to the fed cluster they originate from but the client makes multiple connections in round robin fashion to the configured targets - hence you end up losing some replies.

In theory if we supported priorities/weights in SRV this could probably work but for now no way.

An alternative we could maybe try is supporting the nats.DontRandomize() option, you can then manually configure the target list first listing the one cluster then the other and it should work, mostly, but still not really going to be perfect. But that will also not spread the client connection across multiple brokers which could impact on very large clusters.