What's the state of dcutr?

[MatthiasvB]

Hi everybody! I was wondering if you know the state of dcutr in rust-libp2p? The reason is that it doesn't work very well for me, but I'm not sure if that's because I'm using it wrong, have bad luck with my network environments, or it really just doesn't work well.

I'm aware that a few years back, there has been some effort to evaluate the performance of dcutr, for example by means of punchr. Talks I found on the Internet spoke of hole punching success rates between 90 and 60 %, depending on the network, VPN, etc.

My own success rates, tested by now on three different (semi)private networks, are more in between 1 and 10 % success rates. Is that just me, or do you get similar metrics?

I'm not actually doing much right now. More or less just walking the DHT, connecting to some random (10) relays, and having dcutr active, which does its thing pretty independently.

Any time I do get a dcutr Event, I update statistics of the form

Total successful: 2
Successful peers 2
Total failed: 200
Failed peers 160
Peers with successes and failures: 0

(these are particularly bad)

What I read from that is that the perfomance is, most interestingly, very consistent per peer. If it failed once, it'll fail again. If it's successful once, it'll be successful again (not seen in above stats but in others I got).

Never have I ever had a peer fail and then succeed, or the other way around.

This makes me wonder: Is there room for improvement on that protocol? Are there any plans to tackle that again? At the moment, assuming this is typical performance, it wouldn't be useful for any real world scenario at all.

There also the very wide open gaping hole between the dcutr spec and this popular article on NAT traversal that seems to explain in great detail why dcutr isn't, in fact, adequate.

Should I perhaps move away from dcutr and explore other options, I wonder. I haven't looked at the state of WebRTC in libp2p. Is that going to fix things, or is it a separate topic. Is direct connection between NATed peers high up on the prio list for future development at all?

Really looking forward to hearing your perspectives :)

[drHuangMHT]

Personally I don't really expect dcutr to work very well. I'm hoping for 10% success rate at best but I don't really have the data. I don't think WebRTC will be of any help because it still depends on the physical network, and we can't really do anything about that. Maybe with IPv6?

[elenaf9]

When the punchr project was first developed I ran a rust-client and had a good success rate, roughly matching what was reported in the campaign.
I don't think anything major was changed in the dcutr protocol since then, but I haven't tried it lately and currently don't have a good setup for testing. What type of router/ NATs does your networks have @MatthiasvB? Do you noticed a difference between using QUIC or TCP transport?

[dariusc93]

In my experience, it has been a bit of a hit or miss at times depending on the environment. In a good environment though, it was worked pretty well and had more of a success, although this was using TCP. With QUIC, I would say this can be the same but noticed some flakiness in the earlier days of quic and not sure about the success rate right now at this moment.

[MatthiasvB]

Actually, I hadn't had quic enabled, only TCP. Enabling quic raised the successrate significantly (this is from running multiple days):

Total successful: 6687
Successful peers 811
Total failed: 9980
Failed peers 1013
Peers with successes and failures: 36

So, at the moment, that's a 40% success rate per attempt and a 44% success rate per peer.

Again, a little sad that this "magic" improvement happens without any prior indication that missing quic may be part of the problem.

In the meantime, at least for my home network, I have figured out that it uses a "easy NAT", so no exceptional difficulties expected purely from the networking situation.

I'll probably focus on local network for now to get my application logic working and then see if I find the time to get further into the dcutr debugging

[jxs]

Hi Matthias!

As Elena I also run the rust-client for some time with the success rates matching the reported ones.
We also have the hole punch interop tests running on our CI and currently passing. You can take a look at their implementation here.

So as to make this discussion helpful for future reference

There also the very wide open gaping hole between the dcutr spec and this popular article on NAT traversal that seems to explain in great detail why dcutr isn't, in fact, adequate.

can you elaborate on this? DCUtR was thought with the decentralized nature of libp2p in mind, as you can read more about here. But the fundamental concepts are similar to what tailscale presents on their article.

@drHuangMHT can you elaborate on

Personally I don't really expect dcutr to work very well.

why do you not expect it to work very well?

And @dariusc93 can you elaborate also on

In my experience, it has been a bit of a hit or miss at times depending on the environment. In a good environment though, it was worked pretty well and had more of a success, although this was using TCP.

what in your usage experience is a good environment?

Thank you all!

[drHuangMHT]

@drHuangMHT can you elaborate on

Personally I don't really expect dcutr to work very well.

why do you not expect it to work very well?

That's based on my experience with networks in my country so I don't think it is a very useful piece of information.
But anyway, I've seen zero success when hole-punching with around 10 people, using dcutr and quic over IPv4 connections. That's probably because the ISPs are using symmetric NAT, which is the worst type of NAT for hole-punching.
Maybe it works better in other parts of the world, but not where I live, and it is terrible. That's also why I said it heavily depends on the actual setup, like how the ISPs deal with NAT.

[dariusc93]

And @dariusc93 can you elaborate also on

In my experience, it has been a bit of a hit or miss at times depending on the environment. In a good environment though, it was worked pretty well and had more of a success, although this was using TCP.

what in your usage experience is a good environment?

During the testing and use, the router or modem is one factor. If the NAT is strict, close or configured to block such use cases or the firewall is extremely strict, DCUtR will almost always fail, causing it to fall back to using the relay. Same for double nat, symmetric nat, etc. Another factor is the system firewall on the computer (which also does sometimes impact go-libp2p). Besides that, QUIC sometimes have higher success rate compared to TCP, but QUIC may not always be desirable for some use cases or may have existing issues that may be questionable (ie higher memory usage compared to tcp, especially when one is walking dht, though not sure how it is now these days so ymmv)

[jxs]

Thanks for elaborating folks. As you both referred, hole punching success may be influence by the type of NAT's prevalent on a certain geographic area.
Symmetric NAT's which are a subset of endpoind dependant NAT's are described both in the libp2p blogpost:

There are situations in which hole punching will not work, most notably when one of the nodes is behind a symmetric NAT. In such cases, nodes can instead explicitly add port mappings, either manually or by using UPnP (opens new window).

and the hole punching paper:

NATs using Endpoint-Independent Mapping lend themselves to reliable hole punching. Nodes can rely on their external address
to be stable, and advertise this address to other nodes. On the other hand, there is no reliable way to punch through NATs using Endpoint-Dependent Mappings; the difficulty lies in predicting the port the NAT will assign, which becomes exponentially harder with concurrent connection attempts.

but this is transversal to hole punching in general and not particular to DCUtR as the tailscale article posted by Matthias attests, when direct connection is impossible tailscale resorts to using relayed connections, which is also possible with circuit relay on libp2p world as the libp2p post also states

As a last resort, nodes can leverage external relay (opens new window)nodes.

This to challenge that

There also the very wide open gaping hole between the dcutr spec and this popular article on NAT traversal that seems to explain in great detail why dcutr isn't, in fact, adequate.

is not true, hole punching is adequate for endpoint independent mapping NATs and DCUtR protocol works successfully on that scenario, as the punchr experiment and results showcased.
On endpoint dependent mapping NATs relaying is the consensual solution and the combination of these both techniques should address all the connectivity cases.

[MatthiasvB]

I suppose it depends much on what is considered reasonable effort to obtain direct communication.

While DCUtR may very well conform to the consensual solution of DCUtR, it does not do everything laid out in the tailscale article.

I see DCUtR as a careful way to snipe oneself to a direct connection, while the tailscale article clearly recommends a fire on all fronts out of machine guns approach, if I may paint that picture.

As to your solution of "just use the relays", aren't they heavily rate limited? A few MBs, if I remember correctly? How can I hope to build much more than signalling and such utilities on that?

Sure, I suppose I can run my own relay, have my users authenticate to them and get more bandwidth, but that's added complexity and added cost. If libp2p wants to be the go-to solution for p2p networking, it needs to be able to provide state of the art functionality. I'm sorry, but I don't see that with "DCUtR, relay and done" :/

[dariusc93]

As to your solution of "just use the relays", aren't they heavily rate limited? A few MBs, if I remember correctly? How can I hope to build much more than signalling and such utilities on that?

You can change how much data is used so you could have it unlimited or eset to a specific amount of data you wish the peers to use while using the relay but by default i think its over 200kb (would have to check the code to see)

[jxs]

While DCUtR may very well conform to the consensual solution of DCUtR, it does not do everything laid out in the tailscale article.

again, I don't think this is true. Can you elaborate on what is specifically missing?

I see DCUtR as a careful way to snipe oneself to a direct connection, while the tailscale article clearly recommends a fire on all fronts out of machine guns approach, if I may paint that picture.

It's the same with libp2p, it provides all the tools to do that.

As to your solution of "just use the relays", aren't they heavily rate limited? A few MBs, if I remember correctly? How can I hope to build much more than signalling and such utilities on that?

you can configure that value with max_circuit_bytes

Sure, I suppose I can run my own relay, have my users authenticate to them and get more bandwidth, but that's added complexity and added cost. If libp2p wants to be the go-to solution for p2p networking, it needs to be able to provide state of the art functionality. I'm sorry, but I don't see that with "DCUtR, relay and done" :/

can you elaborate on what were you expecting?
libp2p doesn't want to be anything that it isn't already, a universal set of p2p protocols and different corresponding open source implementations maintained by a group of people that have the aligned interest in having and maintaining that common tech stack.

[MatthiasvB]

Sure, I'll cite a few things from the article which I find relevant to this discussion

We could use a relay that both sides can talk to unimpeded, and have it shuffle packets back and forth. But wait, isn’t that terrible?

Sort of.

But… If you’re not satisfied with “good enough”, there’s still a lot more we can do! What follows is a somewhat miscellaneous set of tricks, which can help us out in specific situations. None of them will solve NAT traversal by itself, but by combining them judiciously, we can get incrementally closer to a 100% success rate.

We can do much better than that, with the help of the birthday paradox. Rather than open 1 port on the hard side and have the easy side try 65,535 possibilities, let’s open, say, 256 ports on the hard side (by having 256 sockets sending to the easy side’s ip:port), and have the easy side probe target ports at random.

Still, with this trick we can make it through a slightly harder network topology than before. That’s a big deal, because home routers tend to be easy NATs, and hard NATs tend to be office routers or cloud NAT gateways. That means this trick buys us improved connectivity for the home-to-office and home-to-cloud scenarios, as well as a few office-to-cloud and cloud-to-cloud scenarios.

This works fine if you only deal in DNS names. If you connect to google.com, turning that into an IP address involves the DNS64 apparatus, which lets the NAT64 get involved without you being any the wiser.

But we care deeply about specific IPs and ports for our NAT and firewall traversal. What about us? If we’re lucky, our device supports CLAT (Customer-side translator — from Customer XLAT). CLAT makes the OS pretend that it has direct IPv4 connectivity, using NAT64 behind the scenes to make it work out. On CLAT devices, we don’t need to do anything special.

CLAT is very common on mobile devices, but very uncommon on desktops, laptops and servers. On those, we have to explicitly do the work CLAT would have done: detect the existence of a NAT64+DNS64 setup, and use it appropriately.

Enter the Interactive Connectivity Establishment (ICE) protocol. Like STUN and TURN, ICE has its roots in the telephony world, and so the RFC is full of SIP and SDP and signalling sessions and dialing and so forth. However, if you push past that, it also specifies a stunningly elegant algorithm for figuring out the best way to get a connection.

Ready? The algorithm is: try everything at once, and pick the best thing that works. That’s it. Isn’t that amazing?

And let's end with this, which is what I hope one day libp2p will provide not just for networking specialists but your general run-off-the-mill application programmer

The good news is that, once you’ve done it, you have something of a superpower: you get to explore the exciting and relatively under-explored world of peer-to-peer applications. So many interesting ideas for decentralized software fall at the first hurdle, when it turns out that talking to each other on the internet is harder than expected. But now you know how to get past that, so go build cool stuff!

and of course, just reading the TL;DR at the end of the article quite concisely enumerates what an "ideal" p2p application should do. I think a lot of these boxes current libp2p can't check.

I do of course understand that implementing all that will be significant effort. I do understand nobody may have time or resources to tackle it. But I don't think a project like libp2p can just say "I'm happy the way I am and ignore that I could be better". People will be looking for the better and a libp2p that doesn't try being it is always going to be a candidate for replacement

[drHuangMHT]

And let's end with this, which is what I hope one day libp2p will provide not just for networking specialists but your general run-off-the-mill application programmer

I personally don't think libp2p is going on the direction of being a one-click solution for connectivity, that should be on the application level instead of library level. Although, to be fair, I want it too. libp2p has been providing various tools to reach that goal, I appreciate that, and yes I would like more tools to be available.

[elenaf9]

If libp2p wants to be the go-to solution for p2p networking, it needs to be able to provide state of the art functionality.

People will be looking for the better and a libp2p that doesn't try being it is always going to be a candidate for replacement

People are free to choose if they want to use libp2p or not depending on whether it suits their use-case. It already provides a large set of application protocols, transport implementations and NAT traversal tools. But of course it doesn't do everything.
Like most other open-source projects it is shaped by what the community and users contribute. If features are missing or could be improved we always appreciate PRs!

@MatthiasvB would you be interested in contributing some of the solutions that you listed above?

[MatthiasvB]

Well, first of all I need to bring my application up to a level where I can see if it would benefit from these additional efforts. They are pretty difficult for limited return, to be fair. If I do decide to try and go that extra mile, I would prefer this logic to be located in libp2p rather than my application. So let's see :)