reboot-coordinator flatcar-linux-update-operator - Pending on new cluster

[clickersmudge]

Hi,
We deploy KubeOne cluster (Hetzner) with one "control plane" and one "worker node".

terraform.tfvars

apiserver_alternative_names = []
cluster_name = "kubeone"
control_plane_replicas = 1
control_plane_type = "cx21"
datacenter = "nbg1"
disable_kubeapi_loadbalancer = false
image = "ubuntu-22.04"
initial_machinedeployment_operating_system_profile = ""
initial_machinedeployment_replicas = 1
ip_range = "192.168.0.0/16"
lb_type = "lb11"
network_zone = "eu-central"
ssh_agent_socket = "env:SSH_AUTH_SOCK"
ssh_port = 22
ssh_private_key_file = ""
ssh_public_key_file = "~/.ssh/id_rsa.pub"
ssh_username = "root"
worker_os = "ubuntu"
worker_type = "cx21"

kubeone.yaml

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster

versions:
  kubernetes: "1.24.8"

clusterNetwork:
  # kube-proxy configurations
  kubeProxy:
    # skipInstallation will skip the installation of kube-proxy
    skipInstallation: true
  # CNI plugin of choice. CNI can not be changed later at upgrade time.
  cni:
    # Only one CNI plugin can be defined at the same time
    # Supported CNI plugins:
    # * canal
    # * weave-net
    # * cilium
    # * external - The CNI plugin can be installed as an addon or manually
    cilium:
      # enableHubble to deploy Hubble relay and UI
      enableHubble: true
      # kubeProxyReplacement defines weather cilium relies on underlying Kernel support to replace kube-proxy functionality by eBPF (strict),
      # or disables a subset of those features so cilium does not bail out if the kernel support is missing (disabled).
      kubeProxyReplacement: "strict"

cloudProvider:
  # Only one cloud provider can be defined at the same time.
  # Possible values:
  # aws: {}
  # azure: {}
  # digitalocean: {}
  # gce: {}
  # hetzner:
  #   networkID: ""
  # openstack: {}
  # equinixmetal: {}
  # vsphere: {}
  # none: {}
  hetzner: {}
  # Set the kubelet flag '--cloud-provider=external' and deploy the external CCM for supported providers
  external: true

features:
  # Configure the CoreDNS deployment
  coreDNS:
    replicas: 3

# Addons are Kubernetes manifests to be deployed after provisioning the cluster
addons:
  enable: true
  # addons is used to enable addons embedded in the KubeOne binary.
  # Currently backups-restic, default-storage-class, and unattended-upgrades are
  # available addons.
  # Check out the documentation to find more information about what are embedded
  # addons and how to use them:
  # https://docs.kubermatic.com/kubeone/v1.5/guides/addons/
  addons:
    # name of the addon to be enabled/deployed (e.g. backups-restic)
    - name: "backups-restic"
      # delete triggers deletion of the deployed addon
      delete: false
      # params is a key-value map of values passed to the addons templating engine,
      # to be used in the addon's manifests. Values defined here override the values
      # defined in globalParams.
      params:
        resticPassword: "YOURS_RESTIC_PASSWORD"
        s3Bucket: "YOURS_S3_BUCKET"
        awsDefaultRegion: ""YOURS_AWS_DEFAULT_REGION"
    # name of the addon to be enabled/deployed (e.g. backups-restic)
    - name: "cluster-autoscaler"
      # delete triggers deletion of the deployed addon
      delete: false
    # name of the addon to be enabled/deployed (e.g. backups-restic)
    - name: "default-storage-class"
      # delete triggers deletion of the deployed addon
      delete: false
    # name of the addon to be enabled/deployed (e.g. backups-restic)
    - name: "unattended-upgrades"
      # delete triggers deletion of the deployed addon
      delete: false

kubectl get pods -A

reboot-coordinator   flatcar-linux-update-operator-755886c688-jkvqm      0/1     Pending             0              20m

kubectl describe pod -n reboot-coordinator   flatcar-linux-update-operator-755886c688-jkvqm
Name:             flatcar-linux-update-operator-755886c688-jkvqm
Namespace:        reboot-coordinator
Priority:         0
Service Account:  flatcar-linux-update-operator-sa
Node:             <none>
Labels:           app=flatcar-linux-update-operator
                  pod-template-hash=755886c688
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/flatcar-linux-update-operator-755886c688
Containers:
  update-operator:
    Image:      quay.io/kinvolk/flatcar-linux-update-operator:v0.7.3
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/update-operator
    Environment:
      POD_NAMESPACE:  reboot-coordinator (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c4cxk (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  kube-api-access-c4cxk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason             Age                 From                Message
  ----     ------             ----                ----                -------
  Warning  FailedScheduling   10m                 default-scheduler   0/3 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Warning  FailedScheduling   5m13s               default-scheduler   0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Warning  FailedScheduling   13s                 default-scheduler   0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Warning  FailedScheduling   30m                 default-scheduler   0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling   24m                 default-scheduler   0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   NotTriggerScaleUp  17m (x61 over 27m)  cluster-autoscaler  pod didn't trigger scale-up:
  Warning  FailedScheduling   14m (x2 over 19m)   default-scheduler   0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Normal   NotTriggerScaleUp  13m                 cluster-autoscaler  pod didn't trigger scale-up:
  Normal   NotTriggerScaleUp  10s (x61 over 10m)  cluster-autoscaler  pod didn't trigger scale-up:

[xmudrii]

As far as I can tell from your terraform.tfvars, your worker nodes are running Ubuntu. flatcar-linux-update-operator is scheduled only in the case you have Flatcar nodes in your cluster, otherwise it'll stay Pending. That being said, what you see here is as expected.

A more technical explanation of this "issue" is that we can't determine what operating system(s) you are using for your worker nodes when deploying the unattended-upgrades addon. Because of that, we deploy all components, but only relevant components for your cluster are scheduled.

[steled]

I had the same problem and created a cleanup manifest, see below.
As described here you need to put the file fluo-cleanup.yaml into the folder ./addons/unattended-upgrades and also update you kubeone.yaml file like below:

In general it's doing a cleanup (deletion) of the fluo.yaml file.

fluo-cleanup.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluo-cleanup
  namespace: kube-system
  labels:
    app: fluo-cleanup
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluo-cleanup
  labels:
    app: fluo-cleanup
rules:
- apiGroups: [""]
  resources: ["namespaces", "serviceaccounts"]
  verbs: ["list", "delete"]
- apiGroups: ["apps"]
  resources: ["daemonsets", "deployments"]
  verbs: ["list", "delete"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]
  verbs: ["list", "delete"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluo-cleanup
  labels:
    app: fluo-cleanup
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluo-cleanup
subjects:
- kind: ServiceAccount
  name: fluo-cleanup
  namespace: kube-system
---
apiVersion: batch/v1
kind: Job
metadata:
  name: fluo-cleanup
  namespace: kube-system
  labels:
    app: fluo-cleanup
spec:
  # ttlSecondsAfterFinished: 60
  # wait for 10 mins and than cleanup pod and job
  ttlSecondsAfterFinished: 600
  template:
    spec:
      containers:
      - command:
        - /bin/bash
        - -c
        - |
          kubectl delete -f https://raw.githubusercontent.com/kubermatic/kubeone/refs/heads/main/addons/unattended-upgrades/fluo.yaml
          kubectl delete serviceaccount,clusterrole -l app=fluo-cleanup -n kube-system
          exit 0
        image: bitnami/kubectl
        imagePullPolicy: IfNotPresent
        name: fluo-cleanup
      restartPolicy: Never
      serviceAccount: fluo-cleanup

kubeone.yaml:

...
addons:
  enable: true
  path: "./addons"
  addons:
    - name: unattended-upgrades

The only thing thats keeping is the ClusterRoleBinding because of "chicken-egg-problem" but I thought that this is the least "problem".