net: MPTCP breaks the existing net.Listener on Linux uses MD5 digest with ENOPROTOOPT (protocol not available)

[YutaroHayakawa]

Go version

1.24.2

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/yutaro.linux/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/yutaro.linux/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2500888817=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/yutaro.linux/go/1.24.2/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/yutaro.linux/go/1.24.2'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/yutaro.linux/.goenv/versions/1.24.2'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/yutaro.linux/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/yutaro.linux/.goenv/versions/1.24.2/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.24.2'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

func TCPMD5SigAvailable() (bool, error) {
        listener, err := net.Listen("tcp", "localhost:0")
        if err != nil {
                return false, fmt.Errorf("listen failed: %w", err)
        }
        defer listener.Close()

        fd, err := listener.(*net.TCPListener).File()
        if err != nil {
                return false, fmt.Errorf("error retrieving socket's file descriptor: %w", err)
        }
        defer fd.Close()

        err = unix.SetsockoptTCPMD5Sig(int(fd.Fd()), unix.IPPROTO_TCP, unix.TCP_MD5SIG, newTcpMD5Sig("1.2.3.4", "key"))
        if err != nil {
                if errors.Is(err, syscall.ENOPROTOOPT) {
                        return false, nil // "protocol not available"
                }
                return false, fmt.Errorf("other error by setting setting socket option: %w", err)
        }
        return true, nil
}

func newTcpMD5Sig(address, key string) *unix.TCPMD5Sig {
        sig := &unix.TCPMD5Sig{}
        addr := net.ParseIP(address)
        if addr.To4() != nil {
                sig.Addr.Family = unix.AF_INET
                copy(sig.Addr.Data[2:], addr.To4())
        } else {
                sig.Addr.Family = unix.AF_INET6
                copy(sig.Addr.Data[6:], addr.To16())
        }
        sig.Keylen = uint16(len(key))
        copy(sig.Key[0:], key)
        return sig
}

func main() {
        available, err := TCPMD5SigAvailable()
        if err != nil {
                fmt.Printf("Error checking TCP_MD5SIG availability: %v\n", err)
                return
        }
        if available {
                fmt.Println("TCP_MD5SIG is available")
        } else {
                fmt.Println("TCP_MD5SIG is not available")
        }
}

Execute above code with sysctl net.mptcp.enabled=0 (MPTCP disabled) and net.mptcp.enabled=1 (MPTCP enabled) on the Linux with CONFIG_TCP_MD5SIG enabled. With 1 it shows TCP_MD5SIG is not available and with 0, it shows TCP_MD5SIG is available.

What did you see happen?

Since v1.24, net.Listen started to enable MPTCP by default for all TCP listeners on Linux.

This makes kernel to call setsockopt of MPTCP (mptcp_setsockopt) instead of TCP.

For some options are compatible with TCP, so they are handled transparently, but in case of MD5, it is not supported in MPTCP, so it returns ENOPROTOOPT here.

This breaks the existing program that uses net.Listen with MD5 digest. The concrete example is BGP speaker such as gobgp. The issue was originally reported for Cilium which internally uses GoBGP (ref).

I'm not entirely sure how we should solve this problem. In theory, this problem occurs for any socket option that is supported in TCP, but not supported in the MPTCP. So far, MD5 digest and TCP_REPAIR are not supported. The former is used in BGP and the latter is used in the container migration tool like CRIU (this is not written in Go though).

The workaround we can make is setting multipathtcp GODEBUG or disable MPTCP with either sysctl or kernel configuration. Both doesn't allow us to adjust the behavior per socket. Since the error happens after socket creation, falling back to the TCP is pretty hard at that point.

(I was overlooking the net.ListenConfig.SetMultipathTCP)

What did you expect to see?

The existing program works without breakage.

[gabyhelp]

Related Issues

• net: MPTCP support is not enabled when apps use `ListenTCP()` or `DialTCP()` #70500 (closed)

Related Code Changes

• net: add initial MPTCP support
• net: mptcp: fallback to TCP in case of any error

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[ianlancetaylor]

CC @matttbe

[matttbe]

Hello,

@YutaroHayakawa thank you for having reported this issue!

@ianlancetaylor thank you for the cc!

In short, TCP MD5 is not compatible with MPTCP, because the MD5 digest in the TCP options is taking 18 bytes + 2 bytes for the padding (if I remember well), out of the 40 bytes in total. There is space left is not enough for MPTCP which typically requires 20 to 28 bytes (depending on the negotiated DSS size) when carrying data. (And by default, TCP Timestamps is used which is 10+2 bytes + eventually SACKS...). For this reason, setsockopt(TCP_MD5SIG) on an MPTCP socket is not supported in the Linux kernel.

But... here we are in a different situation: MPTCP is used by default, and I guess with BGP, the client will not request MPTCP. What we could do, is to support TCP_MD5SIG with an MPTCP socket, which will somehow disable MPTCP support if requested. Would that be a more convenient solution here?

I'm not sure what we could improve in net in Go because, if I remember well, the stream socket (when we decide to use MPTCP) is created when calling net.Listen(), and it is only after that MD5 is requested. @YutaroHayakawa's fix to disable MPTCP in gobgp seems to be the good choice. I guess nobody else is using TCP MD5, it was design for BGP if I'm not mistaken. (and maybe the kernel "fix" is not really needed, but it might be better)

[YutaroHayakawa]

While this is a breaking change, I don't think there's a significant impact TBH. As you mentioned, MD5 is for BGP and BGP speaker written in Go is already rare (GoBGP is the most widely used one which is already fixed), the users rarely use MD5 feature, and there's a direct successor TCP-AO (not sure if MPTCP supports that though). I don't think we need to fix the kernel or Go standard library side just for this use case. BGP side can deal with it. It was a single line fix for GoBGP.

[cherrymui]

cc @neild