valkey-cluster-operator

A Kubernetes Operator to deploy and manage Valkey Clusters.

The main goal of this Kubernetes operator is to provide a simple custom resource definition that can be used to deploy a Valkey cluster in a Kubernetes environment.

The operator provides the following features:

• Scaling up and down of CPU, memory and storage sizes.
• Scaling up and down the number of replicas per shard in the cluster
• Resharding up and down, you can change the number of shards in a cluster and the Operator will handle resharding the slots for you.

What is NOT implemented:

• Valkey version upgrades. Right now only version 8.0.2 of Valkey is supported
• Services. The only way to connect to the Valkey cluster is via the pod IP.

Prerequisites

In order to develop on valkey-cluster-operator, you'll want to install the following tools:

• Kind, used for spinning up a local Kubernetes cluster. This is use to install the operator, as well as to execute e2e tests against the running operator (and the resources it manages)
• Dagger, a task runner that handles most of what Makefile offers, with enhanced caching and use of containers for reproducibility.

Getting Started

valkey-cluster-operator is built using Kube Builder which handles a lot of boilerplate.

For example, some changes you want to make will likely be to the ValkeyCluster Custom Resource Definition (CRD). Those changes are made to api/v1alpha1/valkeycluster_types.go and then Kube Builder handles regenerating the CRD definition found at config/crd/bases/cache.halter.io_valkeyclusters.yaml.

Once your changes are made, you'll want to run the e2e tests which we'll need Dagger for.

All of the functions available via Dagger can be seen like so:

$ dagger functions
▶ connect 0.2s
▶ load module: . 0.5s

Name                           Description
build                          Build the application container
build-and-load-locally         -
build-manager                  Build the application binary
[...]

First, you'll want to build the Go binary for the operator, package it up into a Docker image and load that Docker image into Kind. That sounds a lot but only requires the following step:

$ dagger call build-and-load-locally --sock /var/run/docker.sock
▶ connect 0.2s
▶ load module: . 0.3s
● parsing command line arguments 0.0s

● Host.unixSocket(path: "/var/run/docker.sock"): Socket! 0.0s

● valkeyClusterOperator: ValkeyClusterOperator! 0.0s
▶ .buildAndLoadLocally(
  ┆ source: Host.directory(path: "/Users/cooluser/halter/open-source/platform/valkey-cluster-operator", exclude: [], noCache: true): Directory!
  ┆ sock: Host.unixSocket(path: "/var/run/docker.sock"): Socket!
  ): Void 32.6s

Warning

Make sure that you don't run make test-e2e instead as it will generate binaries specific to your platform (in bin) and will also load the built image into Docker, instead of into your kind cluster.

Once done, you'll want to execute the following command to run the e2e test suite:

$ dagger call e-2-e-test --sock /var/run/docker.sock