Allow the opensearch operator to watch multiple namespaces

[brouberol]

Description

We allow the opensearch-operator to watch multiple namespaces.

We keep the original -watch-namespace flag, to ensure backwards compatibility. We simply split the value over any comma, and populate the cache for each namespace in the csv.

Note: Because the watchNamespace variable was being tested for emptiness before flag.Parse() was being called, it was always empty, causing the operator to always watch all namespaces in the cluster. This is no longer the case.

I have added documentation in the user guide as well as in the chart values.

Because this change occurs in main.go, for which we don't have unit tests, I'll enclose my manual test notes.

Testing

We first rebuild the operator binary.

~/code/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ make build
test -s /Users/brouberol/code/opensearch-k8s-operator/opensearch-operator/bin/controller-gen || GOBIN=/Users/brouberol/code/opensearch-k8s-operator/opensearch-operator/bin go install sigs.k8s.io/controller-tools/cmd/[email protected]
/Users/brouberol/code/opensearch-k8s-operator/opensearch-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go

We ensure that the new behavior is now available.

~/code/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ ./bin/manager --help 2>&1 | grep -A 1 watch-namespace
  -watch-namespace string
    	The comma-separated list of namespaces that the controller manager is restricted to watch. If not set, default is to watch all namespaces.

We run the operator alongside a local minikube.

~/code/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ ./bin/manager -watch-namespace ns1,ns2 
{"level":"info","ts":"2025-09-17T16:34:39.456+0200","logger":"setup","msg":"Starting manager"}
{"level":"info","ts":"2025-09-17T16:34:39.457+0200","msg":"starting server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2025-09-17T16:34:39.457+0200","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
...

We define a namespace-less cluster resource

~/c/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ cat cluster.yaml

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch-cluster
spec:
  general:
    serviceName: opensearch-cluster
    version: '3'
  dashboards:
    enable: true
    version: '3'
    replicas: 1
    resources:
      requests:
        memory: "512Mi"
        cpu: "200m"
      limits:
        memory: "512Mi"
        cpu: "200m"
  nodePools:
    - component: nodes
      replicas: 3
      diskSize: "5Gi"
      nodeSelector:
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
          cpu: "500m"
      roles:
        - "cluster_manager"
        - "data"

We create 3 namespaces

~/c/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ kubectl create namespace ns1
namespace/ns1 created
~/c/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ kubectl create namespace ns2
namespace/ns2 created
~/c/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ kubectl create namespace ns3
namespace/ns3 created

We now create an opensearch cluster in ns1

~/c/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ kubectl create -n ns1 -f cluster.yaml
opensearchcluster.opensearch.opster.io/opensearch-cluster created

We start seeing activity in the operator logs

{"level":"info","ts":"2025-09-17T16:38:19.560+0200","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"ns1"},"namespace":"ns1","name":"opensearch-cluster","reconcileID":"4e9e94b2-8f25-4832-92bf-3a8e23349e3b","cluster":{"name":"opensearch-cluster","namespace":"ns1"}}
{"level":"info","ts":"2025-09-17T16:38:19.566+0200","msg":"Start reconcile - Phase: PENDING","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"ns1"},"namespace":"ns1","name":"opensearch-cluster","reconcileID":"4e9e94b2-8f25-4832-92bf-3a8e23349e3b","cluster":{"name":"opensearch-cluster","namespace":"ns1"}}
...

We now create an opensearch cluster in ns2:

~/c/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ kubectl create -n ns2 -f cluster.yaml
opensearchcluster.opensearch.opster.io/opensearch-cluster created

We start seeing activity in the operator logs, this time related to the cluster in ns2

{"level":"info","ts":"2025-09-17T16:41:40.313+0200","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"ns2"},"namespace":"ns2","name":"opensearch-cluster","reconcileID":"2b0e1d2a-9d9b-4cff-b52c-d3b32789b556","cluster":{"name":"opensearch-cluster","namespace":"ns2"}}
{"level":"info","ts":"2025-09-17T16:41:40.324+0200","msg":"Start reconcile - Phase: PENDING","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"ns2"},"namespace":"ns2","name":"opensearch-cluster","reconcileID":"2b0e1d2a-9d9b-4cff-b52c-d3b32789b556","cluster":{"name":"opensearch-cluster","namespace":"ns2"}}
...

We finally create a cluster in ns3:

~/c/opensearch-k8s-operator/opensearch-operator watch-multiple-ns ?1 ❯ kubectl create -n ns3 -f cluster.yaml
opensearchcluster.opensearch.opster.io/opensearch-cluster created

This time, no log related to the cluster in ns3 is observed in the controller logs.

Chart changes

I render the chart using the default values. The output does not contain the -watch-namespace flag.

~/c/opensearch-k8s-operator/c/opensearch-operator watch-multiple-ns ?1 ❯ helm template  . | grep watch-namespace
~/c/opensearch-k8s-operator/c/opensearch-operator watch-multiple-ns ?1 ❯

I then inject either a single or multiple namespaces to watch, either in a csv or in a list, to ensure that the rendering is correct:

~/c/opensearch-k8s-operator/c/opensearch-operator watch-multiple-ns ?1 ❯ helm template  . --set-json='manager.watchNamespace="ns1"' | grep watch-namespace
        - --watch-namespace=ns1
~/c/opensearch-k8s-operator/c/opensearch-operator watch-multiple-ns ?1 ❯ helm template  . --set-json='manager.watchNamespace="ns1,ns2"' | grep watch-namespace
        - --watch-namespace=ns1,ns2
~/c/opensearch-k8s-operator/c/opensearch-operator watch-multiple-ns ?1 ❯ helm template  . --set-json='manager.watchNamespace=["ns1"]' | grep watch-namespace
        - --watch-namespace=ns1
~/c/opensearch-k8s-operator/c/opensearch-operator watch-multiple-ns ?1 ❯ helm template  . --set-json='manager.watchNamespace=["ns1", "ns2"]' | grep watch-namespace
        - --watch-namespace=ns1,ns2

Issues Resolved

Closes #374

Check List

• Commits are signed per the DCO using --signoff
• Unittest added for the new/changed functionality and all unit tests are successful
• Customer-visible features documented
• No linter warnings (make lint)

If CRDs are changed:

• CRD YAMLs updated (make manifests) and also copied into the helm chart
• Changes to CRDs documented

Please refer to the PR guidelines before submitting this pull request.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[inflatador]

@prudhvigodithi @swoehrl-mw Greetings! I'm an SRE with the Wikimedia Foundation and I work with @brouberol .

We're rolling out a new OpenSearch environment on K8s in the next month or so and I was wondering if y'all had the cycles to review this change? There's more context in our task tracker if y'all are interested.

Thanks for taking a look and feel free to ping here or in OpenSearch Slack if you have any questions or comments.

[prudhvigodithi]

Adding @patelsmit32123 @synhershko to please take a look and add your thoughts.

[synhershko]

We're rolling out a new OpenSearch environment on K8s in the next month or so and I was wondering if y'all had the cycles to review this change? There's more context in our task tracker if y'all are interested.

FWIW we are planning a massive release of a 3.0 version of this operator which will be significantly better and safer to use in production.

[synhershko]

let's not increment versions in a PR please

[synhershko]

What is the use-case for doing what you are doing here? I might be missing something

cc @josedev-union

[brouberol]

What is the use-case for doing what you are doing here?

The use case is mostly deployment convenience (only having to deploy a single operator cluster-wide), as well as aligning with common operator behavior within the Kubernetes ecosystem.

For example, having a single operator being able to watch multiple operators is supported by:

• CloudnativePG
• Spark-operator
• Elasticsearch-operator
• flink-operator
• Strimzi
• redis-operator
• zookeeper-operator
• clickhouse-operator
• stackgres
• grafana operator

Note: We're not actively running all of these operators (only a subset), but I sampled actively maintained operator codebases and documentation to showcase that this is a common behavior.

My point (and IMHO the general sentiment over at #374) is that this behavior is expected by operator users and deployers, as it's become quite standard.

It is something that is natively supported by the operator SDK, and does not take anything away from the current operator behavior, as it only adds the ability to have the operator manage one to many clusters, instead of a single one at the moment.

I hope it clears things up.

Note:: I just realized that for this patch to be complete, it's lacking iterating over watched namespaces to setup the appropriate roles and role bindings in each of them. I'm happy to send that work over if the feature request intention is approved.

[synhershko]

Got it. Happy to merge this once conflicts are resolved and CI is green.