Can CUE be used to prune resources in Kubernetes?

[guettli]

Can CUE prune resources in Kubernetes?

Image my cue code creates 5 manifests today. I update my code now only 4 manifests get created.

How to delete the one manifest which should not exist anymore?

[mvdan]

Yes, you can model Kubernetes deployments/services/etc with CUE: https://github.com/cue-labs/cue-by-example/tree/main/003_kubernetes_tutorial

If you want tighter integration with Kubernetes, there are projects built on CUE which specifically target that ecosystem, such as https://timoni.sh/ or https://holos.run/.

[guettli]

I know that CUE can create Kubernetes manifests. I asked about pruning.

I see that some tools like Timoni can do that.

To be honest, I do not understand the reason why tools like Timoni exist.

You can create and validate yaml with whatever tool you like (bash, cue, kustomize, python, go, ...), and then use a gitops tool to apply the rendered manifests. Aka "Rendered Manifest Pattern".

It is perfectly fine if CUE can not do "prune".

I asked for curiosity, not because I need that feature. It is more the opposite: If CUE does it, then I do not understand why does support it.

[mvdan]

CUE is a general purpose configuration language; it does not have explicit support for k8s or pruning/deleting its resources. Hence why I tried helping by sharing links to projects which target k8s with CUE.

If you want CUE to have a "prune" feature, you'll have to elaborate further what you think that would look like for a general purpose configuration language, because I cannot picture it myself.

[guettli]

Everything is fine. Thank you for your answer. I was just confused because you started the reply with "Yes".

[mvdan]

Ah gotcha, I'm not sure why I did that, re-reading the thread. I was aiming to answer that you can indeed do k8s with just CUE, to some extent.

[aorith]

If you dump the manifests in a folder managed by flux and the configuration for flux has prune enabled then its automatic... I guess that you can do the same with argocd

If you're asking about kubectl commands embedded in *_tool.cue I don't think it's possible.

[slewiskelly]

If you're asking about kubectl commands embedded in *_tool.cue I don't think it's possible.

You could use exec.Run.

Example

-- kubectl_tool.cue --
package kubernetes

import (
	"encoding/yaml"
	"list"
	"tool/exec"
)

command: apply: exec.Run & {
	var: {
		all:      bool | *false  @tag(all,type=bool)
		context:  string | *null @tag(context,type=string)
		dryRun:   bool | *false  @tag(dryrun,type=bool)
		force:    bool | *false  @tag(force,type=bool)
		prune:    bool | *false  @tag(prune,type=bool)
		selector: string | *null @tag(selector,type=string)
	}

	cmd: list.Concat([["kubectl", "apply"], _flags, ["-f", "-"]])

	stdin: yaml.MarshalStream([for r in Resources {r}])

	_flags: [
		if var.all {
			"--all"
		},

		if var.context != null {
			"--context=\(var.context)"
		},

		if var.dryRun {
			"--dry-run=client"
		},

		if var.force {
			"--force"
		},

		if var.prune {
			"--prune"
		},

		if var.selector != null {
			"--selector=\(var.selector)"
		},
	]
}
-- resources.cue --
package kubernetes

Resources: [Name=string]: {
	metadata: {
		name:      "foo"
		namespace: "acme"
		labels: {
			app: "foo"
		}
	}
}

Resources: {
	Deployment: X={
		apiVersion: "apps/v1"
		kind:       "Deployment"

		spec: {
			selector: matchLabels: template.metadata.labels
			template: {
				metadata: X.metadata
				spec: {
					containers: [{
						name:  X.metadata.name
						image: "acme.com/foo:v1.2.3"
						ports: [{
							name:          "http"
							containerPort: 8080
						}]
					}]
				}
			}
		}
	}

	Service: {
		apiVersion: "v1"
		kind:       "Service"

		spec: {
			ports: [{for p in Deployment.spec.template.spec.containers[0].ports {
				name:       p.name
				port:       p.containerPort
				targetPort: p.containerPort
			}}]
			selector: Deployment.metadata.labels
		}
	}

	HorizontalPodAutoscaler: {
		apiVersion: "autoscaling/v2"
		kind:       "HorizontalPodAutoscaler"

		spec: {
			minReplicas: 3
			maxReplicas: 5
			scaleTargetRef: {
				apiVersion: Deployment.apiVersion
				kind:       Deployment.kind
				name:       Deployment.metadata.name
			}
		}
	}
}

#Resource: {
	apiVersion: string
	kind:       string

	metadata: _
	spec:     _
}

[aorith]

Nice! Thanks for the example

[guettli]

@slewiskelly does this really work? For kubectl and prune to work, all resources must have a common label. Otherwise, kubectl can not decide which resources to delete.

[slewiskelly]

Assuming everything is correctly labeled run --prune with --selector:

$ cue export resources.cue -e 'Resources.Deployment.metadata'
{
    "name": "qux",
    "namespace": "acme",
    "labels": {
        "app": "foo"  // Label matches existing resources.
    }
}

$ kubectl get deploy/foo -o yaml | yq '.metadata.labels'
app: foo

$ kubectl get deploy/bar -o yaml | yq '.metadata.labels'
app: bar

$ cue -t dryrun=true -t prune=true -t selector=app=foo cmd apply .
deployment.apps/qux created (dry run)
service/qux created (dry run)
service/foo pruned (dry run)
deployment.apps/foo pruned (dry run)

Or --all can be used, notice the different label selectors cause all other resources to be pruned:

$ cue export resources.cue -e 'Resources.Deployment.metadata'
{
    "name": "qux",
    "namespace": "acme",
    "labels": {
        "app": "qux" // Label doesn't match any existing resources.
    }
}

$ kubectl get deploy/foo -o yaml | yq '.metadata.labels'
app: foo

$ kubectl get deploy/bar -o yaml | yq '.metadata.labels'
app: bar

$ cue -t all=true -t dryrun=true -t prune=true cmd apply .
deployment.apps/qux created (dry run)
service/qux created (dry run)
service/bar pruned (dry run)
service/foo pruned (dry run)
deployment.apps/bar pruned (dry run)
deployment.apps/foo pruned (dry run)

[infogulch]

I wonder if the ideal way to deploy k8s resources with CUE is to export the existing namespace as one key, and evaluate the new "target" namespace from CUE as another key, and diff them. Export new/modified manifests to a directory, export a list of delete commands for manifests in the existing namespace but missing from the CUE config.

[DavidGamba]

Someone mentioned flux and argocd, you can also use kapp, which is a CLI app you could possibly call inside a *_tool.cue.

We have a pattern of having a render.sh script in our apps and the build will call that script to render (with either helm template or cue cmd dump) and then kapp applies the output of that.
You also get a nice diff of the changes.