fix: end2end tests failing due to locked state #1600
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
End-to-end tests were failing while running the chaos testing that deletes random runner pods every 5 seconds.
I was able to reproduce this issue and noticed that sometimes we were getting the state locked during these tests, invalidating their success. This was not happening before PR #1588 was merged.
I've noticed that the terraform-exec library had an upgrade from version v0.15.1-0.20220809152546-4850a69faedb from repo github.com/tf-controller/terraform-exec to v0.17.3 repo github.com/hashicorp/terraform-exec.
After downgrading to the older version, this problem disappeared, and I never got the same locking issues while executing the End-to-End tests.
Tests results
Chaos tests passed with 100 pods being randomly deleted:
