feat: quorum-safe rolling restarts across nodepools #1141
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
josedev-union
requested review from
prudhvigodithi and
synhershko
as code owners
|
@synhershko @prudhvigodithi This PR is WIP but i'd welcome your feedback |
josedev-union
force-pushed
the
safe-rolling-update
branch
from
73330db to
8b5d069
Compare
- implement global candidate selection for restart beyond one sts scope - enforce one deletion per reconcile (maxUnavailable=1) - guarantee one master at a time with cluster-wide quorum checks - keep role-aware path as fallback if no global candidate selected Signed-off-by: josedev-union <[email protected]>
josedev-union
force-pushed
the
safe-rolling-update
branch
from
8b5d069 to
53aa1ef
Compare
synhershko
requested changes
Oct 24, 2025
There was a problem hiding this comment.
The design doc needs updating
Also I'm missing additional safety changes - for example the following at least need to be discussed:
- Do we want to run prechecks before the rolling restart starts, and what should they be? eg do we allow non green clusters to restart? do we require replicas to all indices?
- Do we want to run any checks between node or group restart? eg do we want cluster stabilized and becoming green before proceeding to the next node or group?
- What are failures scenarios in which we stop? what are the options we are going to have to rollback or rescue from failed upgrades?
| @@ -0,0 +1,248 @@ | |||
| # Rolling Restart Improvements for Multi-AZ Master Nodes | |||
There was a problem hiding this comment.
This file describes the issues with the old implementation and suggestions on the new design. Since it's going to be written to docs/designs it really needs to describe the new design without referencing the old way or any existing issues
github-project-automation
bot
moved this to 👀 In Review
in Engineering Effectiveness Board
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Quorum-safe rolling restarts across nodepools
Issues Resolved
#650
#738
Check List
make lint)If CRDs are changed:
make manifests) and also copied into the helm chartPlease refer to the PR guidelines before submitting this pull request.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.