Add NAT64 to enable IPv6 provision in v4 only host #1567

nuhakala · 2025-09-04T09:53:26Z

This adds

Tayga as NAT64 service
Bind9 and configure it as local DNS server with DNS64
IPv6 specific addresses and logic to cluster, controlplane, and worker templates
Updates vars.md with new variables
Updates CNI configuration for IPv6

Why we need this? Previously IPv6 support was added so that the dev env could create bare metal hosts over IPv6. This PR extends that so that we can also provision the bare metal hosts over IPv6 and the provisioned images will be IPv6 only. The tricky part is that our CI environment does not support IPv6 natively, so the VMs cannot access internet over IPv6 and hence also cannot download Kubernetes images and other needed images to set up cluster.

Introducing NAT64/DNS64 solves that issue. It essentially allows the dev env to be deployed with IPv6 only scenario on IPv4 host. Furthermore, this PR introduces required changes to the templates so that they are configured to use IPv6.

NOTE! Running make test does not pass yet. However, these changes should allow IPv6 only BMH to be provisioned with operating system and creating a K8s cluster in those BMHs. Only working with centos node images.

Other not directly related changes that could actually be in their own PR:

Moving the docker daemon template in vm-setup/roles/packages_installation/files/daemon.json to vm-setup/roles/packages_installation/templates/daemon.json to better reflect the purpose of the file.
Fixing kubernetes yum repository address in template.

Other considerations

Currently tests failed after pivoting the control plane components to target cluster. In this step the target cluster needs to create pods for capm3 and other metal3 related components, but this failed because crio inside the target VM was not able to resolve IPv6 address in the registry address. So pulling images failed with cannot parse input: [fd55::1]:5000/localimages/cluster-api-provider-metal3:main. I manually tested this and crio was able to pull images after creating a hostname in /etc/hosts and specifying that hostname instead of bare IPv6 address.
I had to change CLUSTER_APIENDPOINT_HOST into CLUSTER_APIENDPOINT_IP in some templates, because having brackets around the IPv6 address caused errors.
Introducing domain names would solve both these problems, but do we want to spend time on that. Though as this PR introduces proper DNS server, and configures the environment to use that, using local domain names could be feasible.

metal3-io-bot · 2025-09-04T09:53:32Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign smoshiur1237 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nuhakala · 2025-09-04T10:01:26Z

/cc @Rozzii @tuminoid

This PR is quite big so I can also split it into smaller PRs.

The IPv6 is not ready in dev env after this, but I just want to have some feedback and discussion on how to do the DNS stuff.

The bot already pinged @elfosardo but you might also want to test this on your tests, because this is quite big and introduces new moving parts.

nuhakala · 2025-09-04T10:02:47Z

/test metal3-centos-e2e-integration-test-release-1-10 metal3-dev-env-integration-test-ubuntu-main

nuhakala · 2025-09-04T10:05:06Z

tests/roles/run_tests/templates/main/metal3datatemplate-template.yaml

 {% endif %}
 {% if IP_STACK == 'v6' or IP_STACK == 'v4v6' %}
-        - 2001:4860:4860::8888
+        - {{ LOCAL_DNS_V6 }}


This is little controversial. I don't think that it really matters inside the dev env, as it does not really use IPv6 itself, but this might break some users use case.

- Install Tayga as NAT64 service - Install Bind9 and configure it as local DNS server with DNS64 - Add IPv6 specific addresses and logic to cluster, controlplane, and worker templates - Update vars.md with new variables - Update CNI configuration for IPv6 Signed-off-by: Nuutti Hakala <[email protected]>

nuhakala · 2025-09-04T11:09:51Z

/test metal3-centos-e2e-integration-test-release-1-10 metal3-dev-env-integration-test-ubuntu-main

tuminoid · 2025-09-04T12:04:20Z

/cc @terror96

metal3-io-bot · 2025-09-04T12:04:22Z

@tuminoid: GitHub didn't allow me to request PR reviews from the following users: terror96.

Note that only metal3-io members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @terror96

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tuminoid · 2025-09-15T05:53:37Z

/cc @terror96

terror96 · 2025-09-15T13:36:49Z

/cc @terror96

Not much to comment at this point. Need to have a longer look. I just told @nuhakala that it would most likely be beneficial to support cases where NAT64 and DNS64 are deployed separately, because there could be scenarios where one uses a shared DNS64.

terror96

Minor comments & suggestions

terror96 · 2025-09-16T07:23:38Z

tests/roles/run_tests/templates/main/cluster-template-workers-kubeadm-config-centos.yaml

@@ -1,5 +1,7 @@
 # CentOS specific worker kubeadm config
 preKubeadmCommands:
+  - sysctl -w net.ipv6.conf.all.forwarding=1


Would there be need to guard all changes (in this file) to inside {% if IP_STACK == "v6" %}?

terror96 · 2025-09-16T07:38:16Z

tests/roles/run_tests/templates/main/cluster-template-controlplane-kubeadm-config-centos.yaml

@@ -1,5 +1,7 @@
 # CentOS specific controlplane kubeadm config
 preKubeadmCommands:
+  - sysctl -w net.ipv6.conf.all.forwarding=1


Would there be need to guard all changes (in this file) to inside {% if IP_STACK == "v6" %}?

terror96 · 2025-09-16T07:40:07Z

tests/roles/run_tests/templates/main/cluster-template-controlplane-kubeadm-config-centos.yaml

+{# According to VRRP, link local address should come first with v6 #}
+              {{ CLUSTER_APIENDPOINT_LINK_LOCAL }}
+{% endif %}
+              {{ CLUSTER_APIENDPOINT_IP }}


Same as earlier: is there an ...

terror96 · 2025-09-16T07:40:55Z

tests/roles/run_tests/templates/main/cluster-template-cluster.yaml

 spec:
  controlPlaneEndpoint:
-    host: ${ CLUSTER_APIENDPOINT_HOST }
+    host: ${ CLUSTER_APIENDPOINT_IP }


Is there an original setup where we would prefer using CLUSTER_APIENDPOINT_HOST?

terror96 · 2025-09-16T07:45:03Z

tests/roles/run_tests/vars/main.yml

 DOCKER_HUB_PROXY: "{{ lookup('env', 'DOCKER_HUB_PROXY') }}"
 WORKING_DIR: "{{ lookup('env', 'WORKING_DIR') | default('/opt/metal3-dev-env', true) }}"
+LOCAL_DNS_V6: "{{ lookup('env', 'LOCAL_DNS_V6') | default('fd00:abcd::1', true) }}"



How about keeping the original default value: '2001:4860:4860::8888' or is that some obsolete value?

terror96 · 2025-09-16T08:09:05Z

tests/roles/run_tests/vars/main.yml

 CALICO_PATCH_RELEASE: "{{ lookup('env', 'CALICO_PATCH_RELEASE') | default('v3.25.1', true) }}"
 DOCKER_HUB_PROXY: "{{ lookup('env', 'DOCKER_HUB_PROXY') }}"
 WORKING_DIR: "{{ lookup('env', 'WORKING_DIR') | default('/opt/metal3-dev-env', true) }}"
+LOCAL_DNS_V6: "{{ lookup('env', 'LOCAL_DNS_V6') | default('fd00:abcd::1', true) }}"


Traditionally fd ULA-addresses have a 40-bit of Global ID: https://en.wikipedia.org/wiki/Unique_local_address ... yes, it can be zeros, but would still pick something else .... not that it really matters in this case.

terror96 · 2025-09-16T08:11:40Z

vm-setup/roles/nat64/defaults/main.yml

+DNS64_PREFIX: "{{ lookup('env', 'DNS64_PREFIX') | default('fd00:ffff:ffff::/96', true) }}"
+# Local DNS is only used in IPv6 only env
+LOCAL_DNS_V4: "{{ lookup('env', 'LOCAL_DNS_V4') | default('127.0.0.2', true) }}"
+LOCAL_DNS_V6: "{{ lookup('env', 'LOCAL_DNS_V6') | default('fd00:abcd::1', true) }}"


Same ULA nit.

terror96 · 2025-09-16T08:16:21Z

vm-setup/roles/nat64/tasks/teardown_nat.yml

+  shell:
+    cmd: "ip addr del {{ LOCAL_DNS_V6 }}/128 dev lo"
+  become: yes
+  ignore_errors: true


We could also add IPv4 and IPv6 addresses to the tunnel device in order to enable sending of ICMP(v6) messages. It is a stupid that the need to be added manually, because they also need to be specified in the tayga configuration file. Well, life is.

terror96 · 2025-09-16T08:16:52Z

vm-setup/roles/nat64/templates/tayga.conf

+ipv4-addr 192.168.255.1
+prefix {{ DNS64_PREFIX }}
+dynamic-pool 192.168.255.0/24
+data-dir /var/spool/tayga


Maybe also IPv6 address for ICMPv6 traffic.

terror96 · 2025-09-16T08:18:45Z

vm-setup/roles/packages_installation/templates/daemon.json

+    "dns": [ "{{ DOCKER_LOCAL_DNS_V6 }}" ],
+{% endif %}
    "ipv6": {{ DOCKER_IPV6_SUPPORT }},
    "fixed-cidr-v6": "fd00:d0c4::/32",


Not your doing, but I would stick to a minimum of a 48-bit prefix. See: https://en.wikipedia.org/wiki/Unique_local_address

metal3-io-bot · 2025-09-17T20:15:55Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

metal3-io-bot requested a review from dtantsur September 4, 2025 09:53

metal3-io-bot requested a review from elfosardo September 4, 2025 09:53

metal3-io-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 4, 2025

nuhakala force-pushed the nuhakala/ipv6_nat64_and_provision_support branch from 1abf0ce to 6122b0f Compare September 4, 2025 09:55

metal3-io-bot requested review from Rozzii and tuminoid September 4, 2025 10:01

nuhakala commented Sep 4, 2025

View reviewed changes

nuhakala force-pushed the nuhakala/ipv6_nat64_and_provision_support branch from 6122b0f to cfafbd0 Compare September 4, 2025 11:09

metal3-io-bot requested a review from terror96 September 15, 2025 05:53

terror96 reviewed Sep 16, 2025

View reviewed changes

metal3-io-bot added the needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. label Sep 17, 2025

Add NAT64 to enable IPv6 provision in v4 only host #1567

Are you sure you want to change the base?

Add NAT64 to enable IPv6 provision in v4 only host #1567

Uh oh!

Conversation

nuhakala commented Sep 4, 2025

Uh oh!

metal3-io-bot commented Sep 4, 2025

Uh oh!

nuhakala commented Sep 4, 2025

Uh oh!

nuhakala commented Sep 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nuhakala commented Sep 4, 2025

Uh oh!

tuminoid commented Sep 4, 2025

Uh oh!

metal3-io-bot commented Sep 4, 2025

Uh oh!

tuminoid commented Sep 15, 2025

Uh oh!

terror96 commented Sep 15, 2025

Uh oh!

terror96 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

metal3-io-bot commented Sep 17, 2025

Uh oh!

Uh oh!