Linux Network Devices #4538

aojea · 2024-11-21T22:02:46Z

Implementation of opencontainers/runtime-spec#1271

It implements the new proposal to the OCI spec to be able to specify Network Devices that get attached ~~detached~~ from the containers (updated to match the merged proposal opencontainers/runtime-spec#1271)

tests/integration/netdev.bats

kad

LGTM. We are also interested in this use case for our accelerator devices.

libcontainer/network_linux.go

tests/integration/netdev.bats

libcontainer/network_linux.go

libcontainer/configs/validate/validator.go

libcontainer/network_linux.go

libcontainer/factory_linux.go

rata · 2025-02-25T13:05:57Z

@aojea friendly reminder that this should be ready soon, we are cutting 1.3.0-rc.1 soon (maybe this week). The spec part is still not sure when it will be merged?

lifubang · 2025-05-26T10:05:12Z

I'm not familiar with those operations but I'm happy to investigate more if you provide some guidance

diff --git a/tests/integration/checkpoint.bats b/tests/integration/checkpoint.bats
index 3db34061..7608e4ff 100644
--- a/tests/integration/checkpoint.bats
+++ b/tests/integration/checkpoint.bats
@@ -2,14 +2,34 @@
 
 load helpers
 
+function create_netns() {
+       # Create a temporary name for the test network namespace.
+       tmp=$(mktemp -u)
+       ns_name=$(basename "$tmp")
+
+       # Create the network namespace.
+       ip netns add "$ns_name"
+       ns_path=$(ip netns add "$ns_name" 2>&1 | sed -e 's/.*"\(.*\)".*/\1/')
+}
+
+function delete_netns() {
+       # Delete the namespace only if the ns_name variable is set.
+       [ -v ns_name ] && ip netns del "$ns_name"
+}
+
 function setup() {
        # XXX: currently criu require root containers.
        requires criu root
 
        setup_busybox
+
+       # Create a dummy interface to move to the container.
+       ip link add dummy0 type dummy
 }
 
 function teardown() {
+       ip link del dev dummy0
+       delete_netns
        teardown_bundle
 }
 
@@ -100,10 +120,16 @@ function runc_restore_with_pipes() {
 }
 
 function simple_cr() {
+       # Tell runc which network namespace to use.
+       # create_netns
+       # update_config '(.. | select(.type? == "network")) .path |= "'"$ns_path"'"'
+       update_config ' .linux.netDevices |= {"dummy0": {} }'
        runc run -d --console-socket "$CONSOLE_SOCKET" test_busybox
        [ "$status" -eq 0 ]
 
        testcontainer test_busybox running
+       runc exec test_busybox ip address show dev dummy0
+       [ "$status" -eq 0 ]
 
        for _ in $(seq 2); do
                # checkpoint the running container
@@ -119,6 +145,8 @@ function simple_cr() {
 
                # busybox should be back up and running
                testcontainer test_busybox running
+               runc exec test_busybox ip address show dev dummy0
+               [ "$status" -eq 0 ]
        done
 }

aojea · 2025-05-29T09:00:36Z

@lifubang at the cost of duplicating code but to improve test errors troubleshooting I duplicated the test cases so we can have simple_cr and simple_cr_with_netdevice, if there is a problem with the netdevice logic then we can spot it very easy since will only affect ones and no the others

aojea · 2025-05-29T09:16:05Z

failed job with

ssh: Could not resolve hostname localhost: Name or service not known

lifubang · 2025-05-29T09:54:30Z

ci / test (actuated-arm64-6cpu-8gb, 1.24.x, rootless) (pull_request)

@alexellis May I ask your help here, is there some special changes cause we can't use localhost in actuated-arm64-6cpu-8gb? Could you have a suggestion to help us to resolve this issue? 🙏

It seems that the failure occurred from 4 days ago. Please see: https://github.com/opencontainers/runc/actions/runs/15232466902/job/42841958687

The other solution is to change ssh rootless@localhost to ssh [email protected].

lifubang · 2025-06-03T05:25:03Z

The rootless test on the arm64 architecture continues to fail, opened a new issue to track: #4776.

alexellis · 2025-06-03T14:37:02Z

Hi @lifubang happy to help but, we do not provide any support for actuated via GitHub - only by Slack. I've only just seen these mentions.

The other solution is to change ssh rootless@localhost to ssh [email protected].

That sounds like a better solution. I always prefer 127.0.0.1 over "localhost" - especially on systems with IPv6.. sometimes those resolutions will hang indefinitely.

I can't think of a reason why localhost wouldn't resolve off the top of my head, but you can explore the VM image in an SSH session and poke around. That's the best way - access it here - https://docs.actuated.com/tasks/debug-ssh/

You can also create a dummy repo and job and run a command like cat /etc/hosts to see if the entries you expect are present.

Support for rootless containers is built into the Kernel, are there any specific CONFIG_ settings that you typically need?

Alex

alexellis · 2025-06-03T14:45:02Z

What is the context in which this command is being run? On the host directly, in a container?

There is no entry in /etc/hosts if that is being parsed? But I am seeing resolution work.

Can you try a test in your build of adding 127.0.0.1 localhost to your /etc/hosts file for this job?

Perhaps conditionally if needed? Something like this should work

       - name: Add entry to hosts
          if: ${{ runner.arch != 'ARM64' }}
          run: |
             echo "127.0.0.1 localhost" | sudo tee /etc/hosts

runner@87db36ae71e63a0c3a141cd9d597246cd6aaa628:~$ ssh localhost
The authenticity of host '(localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:adNuHUaOs2yQacN8ERU4t5+OzdmE5ONsE9nNzgkPKQw.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

runner@87db36ae71e63a0c3a141cd9d597246cd6aaa628:~$ ping -c1 localhost
PING localhost(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.038 ms

--- localhost ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.038/0.038/0.038/0.000 ms
runner@87db36ae71e63a0c3a141cd9d597246cd6aaa628:~$ nslookup localhost
Server:		127.0.0.53
Address:	127.0.0.53#53

Name:	localhost
Address: 127.0.0.1
Name:	localhost
Address: ::1

runner@87db36ae71e63a0c3a141cd9d597246cd6aaa628:~$ cat /etc/hosts
runner@87db36ae71e63a0c3a141cd9d597246cd6aaa628:~$ cat /etc/hostname
87db36ae71e63a0c3a141cd9d597246cd6aaa628
runner@87db36ae71e63a0c3a141cd9d597246cd6aaa628:~$ curl localhost:80
curl: (7) Failed to connect to localhost port 80 after 0 ms: Connection refused
runner@87db36ae71e63a0c3a141cd9d597246cd6aaa628:~$ curl -4 localhost:80
curl: (7) Failed to connect to localhost port 80 after 0 ms: Connection refused
runner@87db36ae71e63a0c3a141cd9d597246cd6aaa628:~$

If I had 127.0.0.1 localhost, then I see a resolution to IPv4 (exactly what I was describing above)

Seems like you should be more explicit - if you have only bound SSH to 127.0.0.1 over IPv4 - the default is to resolve to the IPv6 loopback, so your workaround is probably the correct solution ssh 127.0.0.1

lifubang · 2025-06-04T09:48:27Z

Thanks, @alexellis!

lifubang · 2025-06-04T09:59:06Z

tests/integration/checkpoint.bats

+	ip address add "$global_ip" dev dummy0
+
+	# Tell runc which network namespace to use.
+	create_netns


If we only use the network namespace created by runc, this test will fail.
I think maybe there is no such scenario, so we can let it to be implemented in the future if someone needs it.

@lifubang can you elaborate? How do you expect this to fail exactly? Also, tests are green, is that unexpected?

@rata Please see:
#4538 (comment)

It's all about the netns managed only by runc, not managed by the high-level container runtimes.
It means that we didn't specific the net ns path in config.json.

alexellis · 2025-06-05T08:19:06Z

I've put 127.0.0.1 first in /etc/hosts so if you'd still like to use ssh localhost - that should work now.

lifubang · 2025-06-05T08:48:46Z

I've put 127.0.0.1 first in /etc/hosts so if you'd still like to use ssh localhost - that should work now.

Thanks, Alex. It's indeed work now, but very strange, I can't see your mentioned change:

+ cat /etc/hosts
localhost 936ea945c6b4019ff071f4088aea0c722b77ddc1
::1 localhost ip6-localhost ip6-loopback
+ cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 1.1.1.1
options edns0

And I see the sshd_config, the config about listener looks like this:

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

So, maybe the ssh listened on all ipv4 and ipv6 now?

Signed-off-by: Antonio Ojea <[email protected]>

Implement support for passing Linux Network Devices to the container network namespace. The network device is passed during the creation of the container, before the process is started. It implements the logic defined in the OCI runtime specification. Signed-off-by: Antonio Ojea <[email protected]>

aojea · 2025-06-18T14:53:01Z

kindly reminder @rata and/or @kolyshkin 😄

aojea · 2025-06-18T15:29:30Z

     exec_test.go:1747: extra fd 25 -> 
    exec_test.go:1750: found 1 extra fds after container.Run
--- FAIL: TestFdLeaks (0.19s)

does not look related

rata · 2025-06-18T16:08:09Z

Yeap, doesn't seem related. I'm triggered a re-run

rata

@aojea Thanks for the PR and the patience with the reviews. LGTM :)

Sorry for the late review, I was AFK.

I'll wait for @lifubang to comment on my question (or if @kolyshkin wants to have a look). I'll aim to merge tomorrow, unless @lifubang opposes. Worst case, we can improve it in a follow-up. Can I count on you @aojea if it's needed? :)

rata · 2025-06-18T14:46:48Z

tests/integration/checkpoint.bats

+	ip address add "$global_ip" dev dummy0
+
+	# Tell runc which network namespace to use.
+	create_netns


@lifubang can you elaborate? How do you expect this to fail exactly? Also, tests are green, is that unexpected?

libcontainer/network_linux.go

aojea · 2025-06-18T17:09:59Z

Can I count on you @aojea if it's needed? :)

@rata I consider myself accountable for all code I add to any project, so you can count on me for maintaining it forever

aojea · 2025-06-19T11:16:20Z

thank you very much folks

aojea mentioned this pull request Nov 21, 2024

Define Linux Network Devices opencontainers/runtime-spec#1271

Merged

aojea force-pushed the netdevices branch 2 times, most recently from bdb31c1 to 0b771ca Compare December 2, 2024 15:28

aojea commented Dec 2, 2024

View reviewed changes

tests/integration/netdev.bats Outdated Show resolved Hide resolved

aojea force-pushed the netdevices branch 2 times, most recently from 07d3b0b to 3833056 Compare December 2, 2024 15:40

kad reviewed Dec 10, 2024

View reviewed changes

lifubang reviewed Dec 10, 2024

View reviewed changes

libcontainer/network_linux.go Outdated Show resolved Hide resolved

tests/integration/netdev.bats Outdated Show resolved Hide resolved

aojea force-pushed the netdevices branch 2 times, most recently from 67f12e0 to d114afe Compare December 12, 2024 07:15

aojea commented Dec 17, 2024

View reviewed changes

libcontainer/network_linux.go Outdated Show resolved Hide resolved

aojea force-pushed the netdevices branch 2 times, most recently from ec90a02 to f4f5d02 Compare December 20, 2024 12:11

aojea force-pushed the netdevices branch 2 times, most recently from 735f9d5 to ce1f612 Compare January 14, 2025 09:47

aojea force-pushed the netdevices branch 4 times, most recently from 6262c5e to c530772 Compare February 6, 2025 09:55

kolyshkin reviewed Feb 7, 2025

View reviewed changes

libcontainer/configs/validate/validator.go Outdated Show resolved Hide resolved

kolyshkin reviewed Feb 7, 2025

View reviewed changes

libcontainer/configs/validate/validator.go Outdated Show resolved Hide resolved

kolyshkin reviewed Feb 7, 2025

View reviewed changes

libcontainer/configs/validate/validator.go Outdated Show resolved Hide resolved

kolyshkin reviewed Feb 7, 2025

View reviewed changes

libcontainer/network_linux.go Outdated Show resolved Hide resolved

kolyshkin reviewed Feb 8, 2025

View reviewed changes

libcontainer/factory_linux.go Outdated Show resolved Hide resolved

aojea force-pushed the netdevices branch 5 times, most recently from 4380e86 to f53d263 Compare February 10, 2025 21:27

aojea force-pushed the netdevices branch from 991b0b1 to f61c32a Compare May 29, 2025 08:59

aojea force-pushed the netdevices branch from f61c32a to c207c53 Compare June 3, 2025 10:30

lifubang reviewed Jun 4, 2025

View reviewed changes

lifubang approved these changes Jun 4, 2025

View reviewed changes

aojea force-pushed the netdevices branch from c207c53 to 89e8bb6 Compare June 9, 2025 16:22

aojea added 3 commits June 18, 2025 15:52

libcontainer/configs package doc

ed5df5f

Signed-off-by: Antonio Ojea <[email protected]>

update runtime-spec

889c7b2

Signed-off-by: Antonio Ojea <[email protected]>

aojea force-pushed the netdevices branch from 89e8bb6 to 8d180e9 Compare June 18, 2025 14:52

rata approved these changes Jun 18, 2025

View reviewed changes

rata merged commit 82fe6e2 into opencontainers:main Jun 19, 2025
44 of 45 checks passed

rata added the impact/changelog label Jun 19, 2025

This was referenced Jun 19, 2025

Add support for specifying Network Devices cncf-tags/container-device-interface#267

Open

Support moving Linux network devices into container netns moby/moby#50251

Open

[Proposal] Support adjust Linux Network Devices containerd/nri#180

Open

Linux Network Devices #4538

Linux Network Devices #4538

Uh oh!

Conversation

aojea commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kad left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rata commented Feb 25, 2025

Uh oh!

lifubang commented May 26, 2025

Uh oh!

aojea commented May 29, 2025

Uh oh!

aojea commented May 29, 2025

Uh oh!

lifubang commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lifubang commented Jun 3, 2025

Uh oh!

alexellis commented Jun 3, 2025

Uh oh!

alexellis commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lifubang commented Jun 4, 2025

Uh oh!

lifubang Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

rata Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

lifubang Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

alexellis commented Jun 5, 2025

Uh oh!

lifubang commented Jun 5, 2025

Uh oh!

aojea commented Jun 18, 2025

Uh oh!

aojea commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rata commented Jun 18, 2025

Uh oh!

rata left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rata Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aojea commented Jun 18, 2025

Uh oh!

Uh oh!

aojea commented Jun 19, 2025

Uh oh!

Uh oh!

aojea commented Nov 21, 2024 •

edited

Loading

lifubang commented May 29, 2025 •

edited

Loading

alexellis commented Jun 3, 2025 •

edited

Loading

aojea commented Jun 18, 2025 •

edited

Loading

rata left a comment •

edited

Loading