Fix #1066: Prevent illegal Pod spec updates on bootstrap Pod

[synhershko]

• Fixed Kubernetes operator error when attempting to update immutable bootstrap Pod spec fields
• Changed bootstrap Pod reconciliation from StatePresent to StateCreated to avoid update attempts on existing Pods

Problem

See #1066

The operator was failing with the following error when trying to update bootstrap Pod specs:
Pod "opensearch-cluster-bootstrap-0" is invalid: spec: Forbidden: pod updates may not change fields other than spec.containers[].image,spec.initContainers[].image,spec.activeDeadlineSeconds,spec.tolerations
(only additions to existing tolerations),spec.terminationGracePeriodSeconds

The issue occurred because the operator was calling ReconcileResource with StatePresent, which attempts to update existing resources to match the desired spec. However, Kubernetes prohibits updating most Pod spec fields after creation, including:

• ServiceAccountName
• NodeName
• Volumes and VolumeMounts
• Tolerations (except additions)
• Affinity

Solution

Changed the bootstrap Pod reconciliation logic in pkg/reconcilers/cluster.go:103 to use reconciler.StateCreated.

This change:

• Uses StateCreated which only creates the Pod if it doesn't exist, avoiding update attempts
• Follows the same pattern used for StatefulSets elsewhere in the codebase (lines 159, 178)
• Preserves correct bootstrap behavior since the Pod is temporary and gets deleted once cluster is initialized

Issues Resolved

#1066

Check List

• Commits are signed per the DCO using --signoff
• Unittest added for the new/changed functionality and all unit tests are successful
• Customer-visible features documented
• No linter warnings (make lint)

If CRDs are changed:

• CRD YAMLs updated (make manifests) and also copied into the helm chart
• Changes to CRDs documented

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[josedev-union]

I’m curious which fields differ between the desired spec and the running pod’s spec. Usually, this happens when the OpenSearchCluster CR is intentionally updated after the bootstrap pod has already been created.

In this PR, it seems those CR changes aren’t being propagated to the bootstrap stage. I can see an edge case where a user sets a wrong or invalid config in the CR, causing the bootstrap to get stuck. If the user then corrects the CR, the fix wouldn’t take effect because the bootstrap pod doesn’t pick up those changes.

Once PR #1120
is merged, we should be safe to delete the bootstrap pod anytime, at least in theory. That means we could recreate it when immutable fields change. We’ll just need to keep a list of immutable fields that actually affect the bootstrap process and recreate the pod only when those fields are modified.

How about your idea? @synhershko

[josedev-union]

plz keep this as Draft