Encode sync rule id in bucket id #333
Merged
+981
−241
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: 5ee6c7b The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
We could also potentially enable this for any new sync rules deploy - in that case we'd just need to persist the state in the sync_rules storage collection/table. But not sure if that's actually worth the extra implementation effort - the approach here would eventually cover the majority of cases as we encourage users to switch to sync streams or the new defaults. |
simolus3
force-pushed
the
fix-timestamp-representation
branch
from
da0954b to
d9a8f65
Compare
simolus3
force-pushed
the
sync-rule-id-in-bucket-ids
branch
from
4eeb885 to
d91c383
Compare
simolus3
force-pushed
the
fix-timestamp-representation
branch
from
32c548a to
1ce5d65
Compare
simolus3
force-pushed
the
sync-rule-id-in-bucket-ids
branch
from
3f41d60 to
db8eff4
Compare
rkistner
reviewed
Aug 27, 2025
rkistner
previously approved these changes
Aug 27, 2025
simolus3
force-pushed
the
sync-rule-id-in-bucket-ids
branch
from
e26d1f5 to
5ee6c7b
Compare
rkistner
approved these changes
Aug 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When deploying new sync rules, the service will re-index everything from a consistent snapshot. Since it re-uses existing bucket ids for this, this means that oplog history checksums within a bucket change too. This is bad for clients, which may download new data ontop of their old oplog without deleting it right away. That increases the sync time (because clients have to wait for a consistent snapshot only to realize that their checksums are off and restart) and can be confusing (because sync progress appears to reach 1.0 before restarting).
A backwards-incompatible change is to encode the id of persisted sync rules in the created bucket ids. This means that every redeploy leads to different bucket ids, causing clients to drop their old buckets instead of trying to consider that data. Since this format would break existing deployments, it's only enabled for sync streams or with an opt-in.