Add Prometheus metrics and certificate rotation #1112

gysel · 2025-09-29T12:24:19Z

Description

Add basic Prometheus metrics to track the state of the managed clusters.
Add flag to automatically rotate generated certificates

This MR could be combined with #1086 to automatically rotate certs without any downtime.

Issues Resolved

Check List

Commits are signed per the DCO using --signoff
Unittest added for the new/changed functionality and all unit tests are successful
Customer-visible features documented
No linter warnings (make lint)

If CRDs are changed:

CRD YAMLs updated (make manifests) and also copied into the helm chart
Changes to CRDs documented

Please refer to the PR guidelines before submitting this pull request.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Michael Gysel <[email protected]>

gysel · 2025-09-29T12:32:24Z

While I'm planning to deploy the current version of the code into pro for our use-case, a bit of polishing is likely needed before this can be merged.

Some of my open questions are:

Should I also expose the CA cert in opensearch_tls_certificate_remaining_days?
Is rotateDaysBeforeExpiry a good name for the flag? Should I enable it by default?
I'm a bit lost with unit tests. Do you think it makes sense to add tests for the certificate renewal?

What else did I miss and should look into?

Signed-off-by: Michael Gysel <[email protected]>

gysel · 2025-09-29T15:24:04Z

Note to myself: I need to review the "TLS" section in the user guide.

synhershko · 2025-09-30T07:03:19Z

Possibly related: #1086

gysel · 2025-09-30T08:07:07Z

Possibly related: #1086

It is definitely related! I'm looking forward to use the hot reloading together with these enhancements.

josedev-union · 2025-10-05T02:18:00Z

opensearch-operator/pkg/reconcilers/tls.go

 	}

 	// Generate node cert, sign it and put it into secret
 	nodeSecret, err := r.client.GetSecret(nodeSecretName, namespace)


Suggested change

nodeSecret, err := r.client.GetSecret(nodeSecretName, namespace)

nodeSecret, err := r.client.GetSecret(nodeSecretName, namespace)

if !k8serrors.IsNotFound(err) {

r.logger.Error(err, "Failed to get secret for transport certificate")

return err

}

Fixed, see e86a879. But I had to add an err != nil && as otherwise the happy-path wouldn't work anymore.

josedev-union · 2025-10-05T02:24:05Z

opensearch-operator/pkg/reconcilers/tls.go

+		if certRenewal {
+			nodeSecret.Data = nodeCert.SecretData(ca)
+		} else {
+			nodeSecret = corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: nodeSecretName, Namespace: namespace}, Type: corev1.SecretTypeTLS, Data: nodeCert.SecretData(ca)}
+			if err := ctrl.SetControllerReference(r.instance, &nodeSecret, r.client.Scheme()); err != nil {
+				return err
+			}
 		}


We can use the same set corev1.Secret regardless of certRenewal

Suggested change

if certRenewal {

nodeSecret.Data = nodeCert.SecretData(ca)

} else {

nodeSecret = corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: nodeSecretName, Namespace: namespace}, Type: corev1.SecretTypeTLS, Data: nodeCert.SecretData(ca)}

if err := ctrl.SetControllerReference(r.instance, &nodeSecret, r.client.Scheme()); err != nil {

return err

}

}

nodeSecret = corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: nodeSecretName, Namespace: namespace}, Type: corev1.SecretTypeTLS, Data: nodeCert.SecretData(ca)}

if !certRenewal {

if err := ctrl.SetControllerReference(r.instance, &nodeSecret, r.client.Scheme()); err != nil {

return err

}

}

Your version is a lot simpler but I need to test whether this correctly retains the ownerReferences on the secret set by .SetControllerReference().

Your code does not retain the owner references in the metadata in a renewal. I now only create a new secret when the certificate is created initially. See 47f02fd

josedev-union · 2025-10-05T02:24:39Z

opensearch-operator/pkg/reconcilers/tls.go

+			}
 		}
+
 		_, err = r.client.CreateSecret(&nodeSecret)


UpdateSecret is needed for renewal case, isn't it?

No, .CreateSecret() internally uses .ReconcileResource() which handles both cases.

Ok, it uses reconciler.StatePresent which handles both cases.

josedev-union · 2025-10-05T02:29:42Z

opensearch-operator/pkg/reconcilers/tls.go

 		}

 		// Generate node cert, sign it and put it into secret
 		nodeSecret, err := r.client.GetSecret(nodeSecretName, namespace)


Only Not Found error is acceptable for going foward the reconciling. I'd check error type earlier and return err if it is another error type than Not Found as I suggested the change above.

Fixed, see e86a879.

josedev-union · 2025-10-05T02:31:33Z

opensearch-operator/config/crd/bases/opensearch.opster.io_opensearchclusters.yaml

+                          rotateDaysBeforeExpiry:
+                            default: -1
+                            description: Automatically rotate certificates before
+                              they expire, set to -1 to disable
+                            type: integer


I prefer enabling this in default. Is there any concern?

Good point - I don't have a strong opinion here. I will definitely enable this in all our deployment. Maybe we could leave it disabled by default for now and enable it by default with the v3.0.0 update?

But that should probably be a decision made by the maintainers.

I assume this change will be merged to main and released as v3.0.
@synhershko wdyt?

josedev-union · 2025-10-05T02:33:10Z

opensearch-operator/pkg/reconcilers/tls.go

+			if certRenewal {
+				nodeSecret.Data = nodeCert.SecretData(ca)
+			} else {
+				nodeSecret = corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: nodeSecretName, Namespace: namespace}, Type: corev1.SecretTypeTLS, Data: nodeCert.SecretData(ca)}
+				if err := ctrl.SetControllerReference(r.instance, &nodeSecret, r.client.Scheme()); err != nil {
+					return err
+				}
 			}
 			_, err = r.client.CreateSecret(&nodeSecret)


Same here.
I'd use UpdateSecret for renewal case.
Also we can use the same code snippet for setting nodeSecret as I suggested changes above.

CreateSecret should be fine. See my comment above.

Signed-off-by: Michael Gysel <[email protected]>

ragu2003 · 2025-10-13T19:41:28Z

opensearch-operator/pkg/helpers/metrics.go

+	ClusterHealth = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "opensearch_cluster_health",
+			Help: "Health status of the cluster. 0=red, 1=yellow, 2=green, -1=unknown",
+		}, []string{
+			"namespace", "opensearch_cluster",
+		})
+	ActiveShards = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "opensearch_cluster_shards_active",
+			Help: "The number of active primary and replica shards.",
+		}, []string{
+			"namespace", "opensearch_cluster",
+		})
+	RelocatingShards = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "opensearch_cluster_shards_relocating",
+			Help: "The number of shards that are currently relocating.",
+		}, []string{
+			"namespace", "opensearch_cluster",
+		})
+	InitializingShards = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "opensearch_cluster_shards_initializing",
+			Help: "The number of shards that are currently initializing.",
+		}, []string{
+			"namespace", "opensearch_cluster",
+		})
+	UnassignedShards = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "opensearch_cluster_shards_unassigned",
+			Help: "The number of shards that are currently unassigned.",


These metrics

opensearch_cluster_health

opensearch_cluster_shards_active

opensearch_cluster_shards_relocating

opensearch_cluster_shards_initializing

opensearch_cluster_shards_unassigned

are already available in opensearch-prometheus-exporter plugin.
(opensearch_cluster_shards_number{cluster="$cluster",type="$shard_type"}
and opensearch_cluster_status{cluster="$cluster"})

opensearch_tls_certificate_remaining_days and opensearch_cluster_info custom metrics will be useful.

To help verifying I just exported the metrics of my current test cluster running OpenSearch 3.2.0 having metrics enabled:

$ curl -k https://localhost:9200/_prometheus/metrics |grep opensearch_cluster_shards_number # TYPE opensearch_cluster_shards_number gauge opensearch_cluster_shards_number{cluster="opensearch-cluster",type="unassigned",} 0.0 opensearch_cluster_shards_number{cluster="opensearch-cluster",type="active",} 159.0 opensearch_cluster_shards_number{cluster="opensearch-cluster",type="relocating",} 0.0 opensearch_cluster_shards_number{cluster="opensearch-cluster",type="initializing",} 0.0 opensearch_cluster_shards_number{cluster="opensearch-cluster",type="active_primary",} 69.0

and

$ curl -k https://localhost:9200/_prometheus/metrics |grep cluster_status opensearch_cluster_status{cluster="opensearch-cluster",} 0.0 # Note: 0 - Green, 1 - Yellow, 2 - Red

Good points! Thanks for the feedback.

These metrics [...] are already available in opensearch-prometheus-exporter plugin.

I tried getting the Prometheus exporter plugin up and running (without the Prometheus operator) and failed to do so as I did not want to use metrics scraping with username/password authentication.
I think that a small set of basic cluster metrics in the Operator makes sense. But the full suite of metrics should definitely be exposed by the exporter.

I'll rename the operator metrics to be clearly distinct from the metrics exposed by the exporter.

I refactored the metrics in 49987b4 and c83d04a.

…'t have name conflicts with the opensearch-prometheus-exporter plugin Switch health codes to be aligned with the exporter codes (0=green, 1=yellow, 2=red, -1=unknown) Signed-off-by: Michael Gysel <[email protected]>

Signed-off-by: Michael Gysel <[email protected]>

josedev-union

Please resolve the conflicts

gysel added 3 commits September 29, 2025 14:18

Add metrics and rotate managed certificates

1205a9b

Signed-off-by: Michael Gysel <[email protected]>

Update manifests

34ff834

Signed-off-by: Michael Gysel <[email protected]>

Review docs

95547a7

Signed-off-by: Michael Gysel <[email protected]>

gysel requested review from idanl21, jochenkressin, pchmielnik, prudhvigodithi, salyh and swoehrl-mw as code owners September 29, 2025 12:24

github-project-automation bot added this to Engineering Effectiveness Board Sep 29, 2025

Remove commented code

3bc8f19

Signed-off-by: Michael Gysel <[email protected]>

gysel added 2 commits September 29, 2025 17:03

Use time.Until (fix linter finding)

32725e7

Signed-off-by: Michael Gysel <[email protected]>

Directly reference embedded field (fix linter finding)

53995d9

Signed-off-by: Michael Gysel <[email protected]>

josedev-union suggested changes Oct 6, 2025

View reviewed changes

github-project-automation bot moved this to 👀 In Review in Engineering Effectiveness Board Oct 6, 2025

gysel added 3 commits October 10, 2025 13:54

Improve error handling

e86a879

Signed-off-by: Michael Gysel <[email protected]>

Simplify secret update code

6a35622

Signed-off-by: Michael Gysel <[email protected]>

Retain owner references when renewing the certs

47f02fd

Signed-off-by: Michael Gysel <[email protected]>

gysel requested a review from josedev-union October 13, 2025 09:35

ragu2003 reviewed Oct 13, 2025

View reviewed changes

gysel added 2 commits October 15, 2025 09:05

Consolidate all shared metrics into opensearch_operator_cluster_shards

c83d04a

Signed-off-by: Michael Gysel <[email protected]>

josedev-union reviewed Oct 27, 2025

View reviewed changes

Uh oh!

Add Prometheus metrics and certificate rotation #1112

Are you sure you want to change the base?

Add Prometheus metrics and certificate rotation #1112

Conversation

gysel commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

gysel commented Sep 29, 2025

Uh oh!

gysel commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

synhershko commented Sep 30, 2025

Uh oh!

gysel commented Sep 30, 2025

Uh oh!

josedev-union Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gysel Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gysel Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josedev-union left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gysel commented Sep 29, 2025 •

edited

Loading

gysel commented Sep 29, 2025 •

edited

Loading

josedev-union Oct 5, 2025 •

edited

Loading

gysel Oct 13, 2025 •

edited

Loading

gysel Oct 10, 2025 •

edited

Loading