config: Individually configure rollover size for OpenSearch indices

[linus-elastisys]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

What does this PR do / why do we need this PR?

This allows the setting of individual rollover settings for the different indices in OpenSearch. Previously a 5 GB rollover was used for all indices, even though Prometheus alert rules were set to different values. This PR introduces individual settings for the kubernetes, kubeaudit, other and authlog indices. The new default rollover values are based on the existing alerts sizes (slightly smaller).

• Fixes Rework index size alerts and rollover thresholds in OpenSearch #2644

Information to reviewers

The authlog alert size was slightly increased because it felt too tiny to set a rollover size to 1 MB, and leaving both alert and rollover at 2 MB felt like it could cause false alerts.

One caveat is that the new ISM policy (with smaller size) is only applied after the next rollover, which means that the old size will still be in effect for up to a day.

Question to reviewers: Should this be a kind/admin-change and/or have a "Platform Administrator notice" since it changes the configuration file and schema? I'm not sure if my changes here will require any action to be taken by platform admins. Also not sure if this is something that requires a migration.

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change updates CRDs
  • The change updates the config and the schema
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[linus-elastisys]

I tested this by manually submitting a lot of stuff to the authlog to fill it up above the default 2 MB limit.

Before: we can see that the 2 MB limit is there.

After posting a lot of example documents using the API, there's a successful rollover of the -10 index, and a new -11 index is created when it's detected (seems like ISM policies are checked every ~5 minutes, so some overrun of the limit is expected):

[elastisys-staffan]

Tested installation on my dev cluster, works as expected. Nice work! 🎉

[aarnq]

LGTM, I have a suggestion though.

[lunkan93]

Nice work 👍