fix: fix reverse routing for KubeSpan #10074

dsseng · 2024-12-30T10:26:06Z

This allows it to not come down when rp_filter is enabled.
Fixes #9814

Signed-off-by: Dmitry Sharshakov [email protected]

        chain kubespan_prerouting { # handle 50
                type filter hook prerouting priority filter; policy accept;
                meta mark & 0x00000060 == 0x00000020 accept # handle 51
                ip daddr { 172.20.0.3-172.20.0.4 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 53
                ip6 daddr { fdbd:f075:61fa:5502:2c26:e0ff:fe99:e2a4, fdbd:f075:61fa:5502:e88a:18ff:fed2:32f3 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 55
                meta mark & 0x00000040 == 0x00000040 ip saddr != { 172.20.0.3-172.20.0.4 } meta mark set meta mark ^ 0x00000040 accept # handle 57
                meta mark & 0x00000040 == 0x00000040 ip6 saddr != { fdbd:f075:61fa:5502:2c26:e0ff:fe99:e2a4, fdbd:f075:61fa:5502:e88a:18ff:fed2:32f3 } meta mark set meta mark ^ 0x00000040 accept # handle 59
                iifname "kubespan" meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 60
        }

        chain kubespan_prerouting { # handle 50
                type filter hook prerouting priority filter; policy accept;
                meta mark & 0x00000060 == 0x00000020 accept # handle 51
                ip daddr { 172.20.0.2, 172.20.0.4 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 53
                ip6 daddr { fdbd:f075:61fa:5502:5cd0:eff:fea3:252f, fdbd:f075:61fa:5502:e88a:18ff:fed2:32f3 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 55
                meta mark & 0x00000040 == 0x00000040 ip saddr != { 172.20.0.2, 172.20.0.4 } meta mark set meta mark ^ 0x00000040 accept # handle 57
                meta mark & 0x00000040 == 0x00000040 ip6 saddr != { fdbd:f075:61fa:5502:5cd0:eff:fea3:252f, fdbd:f075:61fa:5502:e88a:18ff:fed2:32f3 } meta mark set meta mark ^ 0x00000040 accept # handle 59
                iifname "kubespan" meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 60
        }

dsseng · 2024-12-30T10:31:48Z

internal/app/machined/pkg/controllers/runtime/kernel_param_defaults.go

+		// Consider fwmark for rp_filter routing table lookup.
+		{
+			Key:   "proc.sys.net.ipv4.conf.kubespan.src_valid_mark",
+			Value: "1",
+		},


We might even consider enabling rp_filter on all interfaces by default as Red Hat and SUSE do for increased security against IP spoofing

I guess we can set to 1 as default like RHEL

Let's discuss that later for 1.10, it's not to be backported like this change is

frezbo · 2024-12-30T11:39:29Z

internal/app/machined/pkg/controllers/runtime/kernel_param_defaults.go

 		},
+		// Consider fwmark for rp_filter routing table lookup.
+		{
+			Key:   "proc.sys.net.ipv4.conf.kubespan.src_valid_mark",


shouldn't this be rp_filter instead of src_valid_mark 🤔 ?

rp_filter will get overriden. This is what I mean:

src_valid_mark - BOOLEAN 0 - The fwmark of the packet is not included in reverse path route lookup. This allows for asymmetric routing configurations utilizing the fwmark in only one direction, e.g., transparent proxying. 1 - The fwmark of the packet is included in reverse path route lookup. This permits rp_filter to function when the fwmark is used for routing traffic in both directions. This setting also affects the utilization of fmwark when performing source address selection for ICMP replies, or determining addresses stored for the IPOPT_TS_TSANDADDR and IPOPT_RR IP options. The max value from conf/{all,interface}/src_valid_mark is used. Default value is 0.

https://docs.kernel.org/networking/ip-sysctl.html

It's not to disable rp_filter (you cannot override someone enabling rp_filter in mode 2 (loose), like current Kube-Router versions), but to make it honor policy-based routing. To avoid potential side effects we only do this on kubespan interface as it's the interface we do our policy-based routing on.

okay, that makes sense 👍

internal/app/machined/pkg/controllers/runtime/kernel_param_defaults.go

internal/app/machined/pkg/controllers/kubespan/manager.go

dsseng · 2025-01-28T16:47:58Z

internal/app/machined/pkg/controllers/kubespan/manager.go

+							Xor:  constants.KubeSpanDefaultForceFirewallMark,
+						},
+						Verdict: pointer.To(nethelpers.VerdictAccept),
+					},


Full rule list in the modified chain for review:

chain kubespan_prerouting { # handle 50 type filter hook prerouting priority filter; policy accept; meta mark & 0x00000060 == 0x00000020 accept # handle 51 ip daddr { 172.20.0.3-172.20.0.4 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 53 ip6 daddr { fdbd:f075:61fa:5502:2c26:e0ff:fe99:e2a4, fdbd:f075:61fa:5502:e88a:18ff:fed2:32f3 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 55 meta mark & 0x00000040 == 0x00000040 ip saddr != { 172.20.0.3-172.20.0.4 } meta mark set meta mark ^ 0x00000040 accept # handle 57 meta mark & 0x00000040 == 0x00000040 ip6 saddr != { fdbd:f075:61fa:5502:2c26:e0ff:fe99:e2a4, fdbd:f075:61fa:5502:e88a:18ff:fed2:32f3 } meta mark set meta mark ^ 0x00000040 accept # handle 59 iifname "kubespan" meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 60 }

chain kubespan_prerouting { # handle 50 type filter hook prerouting priority filter; policy accept; meta mark & 0x00000060 == 0x00000020 accept # handle 51 ip daddr { 172.20.0.2, 172.20.0.4 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 53 ip6 daddr { fdbd:f075:61fa:5502:5cd0:eff:fea3:252f, fdbd:f075:61fa:5502:e88a:18ff:fed2:32f3 } meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 55 meta mark & 0x00000040 == 0x00000040 ip saddr != { 172.20.0.2, 172.20.0.4 } meta mark set meta mark ^ 0x00000040 accept # handle 57 meta mark & 0x00000040 == 0x00000040 ip6 saddr != { fdbd:f075:61fa:5502:5cd0:eff:fea3:252f, fdbd:f075:61fa:5502:e88a:18ff:fed2:32f3 } meta mark set meta mark ^ 0x00000040 accept # handle 59 iifname "kubespan" meta mark set meta mark & 0xffffffdf | 0x00000040 accept # handle 60 }

okay, now with kube-router CNI (and coredns) doesn't come up

internal/app/machined/pkg/controllers/kubespan/manager.go

This allows it to not come down when rp_filter is enabled. Fixes siderolabs#9814 Co-authored-by: Andrey Smirnov <[email protected]> Signed-off-by: Dmitry Sharshakov <[email protected]>

github-actions · 2025-03-21T02:07:34Z

This PR is stale because it has been open 45 days with no activity.

rothgar · 2025-06-26T00:32:11Z

Did this get fixed?

smira · 2025-06-26T17:13:38Z

Did this get fixed?

kube-router was updated to be less agressive about rpfilter, so it "just works" now. this PR hasn't been finished.

rothgar · 2025-06-26T20:18:59Z

If it works with kube-router now is this PR still needed?

smira · 2025-06-27T09:32:58Z

If it works with kube-router now is this PR still needed?

yes, otherwise we would have closed it

enabling rp_filter by default would be nice as a feature

stevefan1999-personal · 2025-07-01T03:57:24Z

maybe this could fix #11244 idk

github-actions · 2025-08-16T02:13:56Z

This PR is stale because it has been open 45 days with no activity.

stevefan1999-personal · 2025-08-16T16:36:51Z

no stale

github-actions · 2025-10-02T02:04:56Z

This PR is stale because it has been open 45 days with no activity.

stevefan1999-personal · 2025-10-02T03:12:23Z

no stale

talos-bot added the status/ok-to-test label Dec 30, 2024

dsseng mentioned this pull request Dec 30, 2024

fix(DSR): setup DSR inside pod on local eps only cloudnativelabs/kube-router#1651

Merged

dsseng commented Dec 30, 2024

View reviewed changes

frezbo reviewed Dec 30, 2024

View reviewed changes

dsseng force-pushed the kubespan-reverse branch 2 times, most recently from f33df6b to cbb11f3 Compare December 30, 2024 20:23

smira reviewed Jan 8, 2025

View reviewed changes

internal/app/machined/pkg/controllers/runtime/kernel_param_defaults.go Outdated Show resolved Hide resolved

dsseng force-pushed the kubespan-reverse branch from cbb11f3 to 457d1d4 Compare January 14, 2025 20:23

dsseng commented Jan 14, 2025

View reviewed changes

internal/app/machined/pkg/controllers/kubespan/manager.go Show resolved Hide resolved

dsseng commented Jan 15, 2025

View reviewed changes

internal/app/machined/pkg/controllers/kubespan/manager.go Outdated Show resolved Hide resolved

dsseng added the area/kubespan label Jan 15, 2025

smira added the integration/misc-1 label Jan 15, 2025

dsseng force-pushed the kubespan-reverse branch from c1ad398 to 5f44463 Compare January 15, 2025 17:23

dsseng force-pushed the kubespan-reverse branch from 5f44463 to 61ff3d1 Compare January 28, 2025 16:46

dsseng added the integration/cilium label Jan 28, 2025

dsseng commented Jan 28, 2025

View reviewed changes

smira reviewed Jan 28, 2025

View reviewed changes

internal/app/machined/pkg/controllers/kubespan/manager.go Show resolved Hide resolved

fix: fix reverse routing for KubeSpan

c93d867

This allows it to not come down when rp_filter is enabled. Fixes siderolabs#9814 Co-authored-by: Andrey Smirnov <[email protected]> Signed-off-by: Dmitry Sharshakov <[email protected]>

dsseng force-pushed the kubespan-reverse branch from 61ff3d1 to c93d867 Compare February 3, 2025 20:17

github-actions bot added the Stale label Mar 21, 2025

smira removed this from Backports to v1.9 May 1, 2025

github-actions bot removed the Stale label Jun 26, 2025

github-actions bot added the Stale label Aug 16, 2025

github-actions bot removed the Stale label Aug 17, 2025

github-actions bot added the Stale label Oct 2, 2025

github-actions bot removed the Stale label Oct 3, 2025

Uh oh!

fix: fix reverse routing for KubeSpan #10074

Are you sure you want to change the base?

fix: fix reverse routing for KubeSpan #10074

Uh oh!

Conversation

dsseng commented Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Mar 21, 2025

Uh oh!

rothgar commented Jun 26, 2025

Uh oh!

smira commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rothgar commented Jun 26, 2025

Uh oh!

smira commented Jun 27, 2025

Uh oh!

stevefan1999-personal commented Jul 1, 2025

Uh oh!

github-actions bot commented Aug 16, 2025

Uh oh!

stevefan1999-personal commented Aug 16, 2025

Uh oh!

github-actions bot commented Oct 2, 2025

Uh oh!

stevefan1999-personal commented Oct 2, 2025

Uh oh!

Uh oh!

dsseng commented Dec 30, 2024 •

edited

Loading

smira commented Jun 26, 2025 •

edited

Loading