[BUG] useRoleBindings=true in operator v2.8.0 causes cluster-scoped resource watch/list failures

[chris-cherian]

What is the bug?

When deploying OpenSearch Operator v2.8.0 with useRoleBindings: true and manager.watchNamespace: , the operator logs errors indicating it's trying to list or watch cluster-scoped resources, even though I intended it to run namespace-scoped. This breaks the operator (it fails to watch CRDs and Kubernetes resources).

How can one reproduce the bug?

1. Deploy operator Helm chart v2.8.0 with values including:

useRoleBindings: true
manager:
watchNamespace: opensearch

2. Ensure the ServiceAccount has only Role + RoleBinding (namespaced) permissions.

3. Observe operator logs such as:

“Failed to watch” … cannot list resource "statefulsets" in API group "apps" at the cluster scope
cannot list resource "opensearchactiongroups.opensearch.opster.io" … at the cluster scope
cannot list resource "secrets" … at the cluster scope
etc.

What is the expected behavior?

When useRoleBindings: true is enabled and watchNamespace is set, the operator should run with only namespace-scoped permissions (Roles & RoleBindings), and should not try to access or watch cluster-scoped resources that require ClusterRole permissions.

What is your host/environment?

Node OS: Ubuntu 22
Kubernetes version: 1.31

Do you have any additional context?

0918 06:22:56.093117 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchActionGroup: opensearchactiongroups.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchactiongroups\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchActionGroup" E0918 06:22:56.148462 1 reflector.go:200] "Failed to watch" err="failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"statefulsets\" in API group \"apps\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.StatefulSet" E0918 06:22:56.381891 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchIndexTemplate: opensearchindextemplates.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchindextemplates\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchIndexTemplate" E0918 06:22:56.952051 1 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"secrets\" in API group \"\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.Secret" E0918 06:22:57.266743 1 reflector.go:200] "Failed to watch" err="failed to list *v1.Deployment: deployments.apps is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"deployments\" in API group \"apps\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.Deployment" E0918 06:22:57.287984 1 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"configmaps\" in API group \"\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.ConfigMap" E0918 06:22:57.467343 1 reflector.go:200] "Failed to watch" err="failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"pods\" in API group \"\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.Pod" E0918 06:22:57.534592 1 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"services\" in API group \"\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.Service" E0918 06:22:57.651691 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchroles\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchRole" E0918 06:22:58.067034 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpenSearchISMPolicy: opensearchismpolicies.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchismpolicies\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpenSearchISMPolicy" E0918 06:22:58.261374 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchComponentTemplate: opensearchcomponenttemplates.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchcomponenttemplates\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchComponentTemplate" E0918 06:23:07.656269 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpenSearchCluster: opensearchclusters.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchclusters\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpenSearchCluster" E0918 06:23:10.616130 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchUserRoleBinding: opensearchuserrolebindings.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchuserrolebindings\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchUserRoleBinding" E0918 06:23:11.545951 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpenSearchISMPolicy: opensearchismpolicies.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchismpolicies\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpenSearchISMPolicy" E0918 06:23:12.462633 1 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"services\" in API group \"\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.Service" E0918 06:23:12.722120 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchTenant: opensearchtenants.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchtenants\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchTenant" E0918 06:23:13.689099 1 reflector.go:200] "Failed to watch" err="failed to list *v1.Deployment: deployments.apps is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"deployments\" in API group \"apps\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.Deployment" E0918 06:23:13.917150 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchSnapshotPolicy: opensearchsnapshotpolicies.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchsnapshotpolicies\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchSnapshotPolicy" E0918 06:23:14.568096 1 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"configmaps\" in API group \"\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.ConfigMap" E0918 06:23:15.624326 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchUser: opensearchusers.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchusers\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchUser" E0918 06:23:17.018432 1 reflector.go:200] "Failed to watch" err="failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"statefulsets\" in API group \"apps\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.StatefulSet" E0918 06:23:18.191364 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchRole: opensearchroles.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchroles\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchRole" E0918 06:23:20.671829 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchIndexTemplate: opensearchindextemplates.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchindextemplates\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchIndexTemplate" E0918 06:23:20.798836 1 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"secrets\" in API group \"\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.Secret" E0918 06:23:21.328259 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchActionGroup: opensearchactiongroups.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchactiongroups\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchActionGroup" E0918 06:23:22.971965 1 reflector.go:200] "Failed to watch" err="failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"pods\" in API group \"\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.Pod" E0918 06:23:23.515594 1 reflector.go:200] "Failed to watch" err="failed to list *v1.OpensearchComponentTemplate: opensearchcomponenttemplates.opensearch.opster.io is forbidden: User \"system:serviceaccount:opensearch:opensearch-operator-controller-manager\" cannot list resource \"opensearchcomponenttemplates\" in API group \"opensearch.opster.io\" at the cluster scope: Azure does not have opinion for this user." logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.33.1/tools/cache/reflector.go:285" type="*v1.OpensearchComponentTemplate"

[chris-cherian]

I believe this feature was introduced in #831

[chris-cherian]

I’m trying to run separate operators in different namespaces, with each operator only watching its own specific namespace. Is this behavior supported in operator version 2.8.0?

Below are the values.yaml spec changes I’ve applied to the operator:

manager:
  watchNamespace: opensearch

useRoleBindings: true

[inflatador]

We've been affected by this bug at Wikimedia Foundation as well. I believe this change submitted by my colleague @brouberol could also fix the problem, as he mentions

Note: Because the watchNamespace variable was being tested for emptiness before flag.Parse() was being called, it was always empty, causing the operator to always watch all namespaces in the cluster. This is no longer the case.

If any operator devs are available to take a look, it would be greatly appreciated.