Implement retry flaky steps #684

nhv96 · 2025-03-29T13:56:35Z

🤔 What's changed?

Added support for retrying flaky steps by returning ErrRetry
Added CLI option for MaxRetries attempts
Added example of the feature

⚡️ What's your motivation?

Discussed here.

🏷️ What kind of change is this?

⚡ New feature (non-breaking change which adds new behaviour)

♻️ Anything particular you want feedback on?

📋 Checklist:

I agree to respect and uphold the Cucumber Community Code of Conduct
I've changed the behaviour of the code
- I have added/updated tests to cover my changes.
My change requires a change to the documentation.
- I have updated the documentation accordingly.
Users should know about my change
- I have added an entry to the "Unreleased" section of the CHANGELOG, linking to this pull request.

This text was originally generated from a template, then edited by hand. You can modify the template here.

* retry failed pickles * revert some changes * use example steps from cck * update example * implement retry with godog.ErrRetry * unit test for retry flaky * Update flaky feature description * add --retry to CLI option * update test * update test for --retry cli option * update test * handle err without printing retry steps * storage methods to upsert step result * rename wording * revert change * update feature test (will remove this feature) * revert unused files * revert unused files * improve readability * remove unused code * cleanup

* retry failed pickles * revert some changes * use example steps from cck * update example * implement retry with godog.ErrRetry * unit test for retry flaky * Update flaky feature description * add --retry to CLI option * update test * update test for --retry cli option * update test * handle err without printing retry steps * storage methods to upsert step result * rename wording * revert change * update feature test (will remove this feature) * revert unused files * revert unused files * improve readability * remove unused code * cleanup * update changelog

luke-hill

Some minor fixes suggested. However a larger question is, are you seeking conformance with standard cucumber execution / cck conformance (I know you might not be there yet).

The reason why, is that a lot of your implementation details surrounding the specific status is unique and not what all standard implementors do

luke-hill · 2025-03-31T14:28:52Z

_examples/retry-flaky/godogs_test.go

+	sc.Step(`^a step that passes the second time`, func(ctx context.Context) (context.Context, error) {
+		secondTimePass++
+		if secondTimePass < 2 {
+			return ctx, fmt.Errorf("unexpected network connection, %w", godog.ErrRetry)


For stuff that will inevitably become part of living documentation I would make this more informative than obfuscating.

See https://github.com/cucumber/compatibility-kit/blob/main/devkit/samples/retry/retry.feature.ts (Which in itself probably could be a bit better)

Right, I think I should just return godog.ErrRetry instead.

luke-hill · 2025-03-31T14:29:32Z

_examples/retry-flaky/godogs_test.go

+		return ctx, nil
+	})
+
+	fifthTimePass := 0


This should be removed as we want this always to fail. The conditional below should also be removed. It should always return the formatted error

Yes I agree, this folder is for examples, I should not add this case here (this case is supposed to be in unit test over here only).

luke-hill · 2025-03-31T14:31:14Z

internal/flags/flags.go

@@ -46,4 +46,6 @@ built-in formatters are:
 specify SEED to reproduce the shuffling from a previous run
  --random=5738`)
 	flagSet.Lookup(prefix + "random").NoOptDefVal = "-1"
+
+	flagSet.IntVar(&opts.MaxRetries, prefix+"retry", opts.MaxRetries, "retry n times when a step encounter error")


Suggested change

flagSet.IntVar(&opts.MaxRetries, prefix+"retry", opts.MaxRetries, "retry n times when a step encounter error")

flagSet.IntVar(&opts.MaxRetries, prefix+"retry", opts.MaxRetries, "Specify the number of times to retry failing tests (default: 0)")

Make sure this is defaulting to 0

If you want to see where the original config parser is set with all of the messages check here: https://github.com/cucumber/cucumber-ruby/blob/main/lib/cucumber/cli/options.rb

Yes it will be default 0. I'll update the message.

luke-hill · 2025-03-31T14:33:23Z

internal/flags/flags_test.go

@@ -51,6 +53,7 @@ func Test_BindFlagsShouldRespectFlagOverrides(t *testing.T) {
 		"--optOverrides.strict=false",
 		"--optOverrides.no-colors=false",
 		"--optOverrides.random=2",
+		"--optOverrides.retry=3",


I'm not sure I agree with this assertion or infact the one above saying concurrency is 3. Are you taking your options parser and then adding whatever values you input? We want this to be a single source of truth, not something additive.

If I'm misunderstanding, feel free to ignore.

Ah this is not concurrency option. In this test MaxRetries is set to 1 when initialize the option object and then we pass in CLI flags to override the option value, the goal is to make sure that CLI options work as expected.

luke-hill · 2025-03-31T14:34:17Z

internal/flags/flags_test.go

@@ -61,4 +64,5 @@ func Test_BindFlagsShouldRespectFlagOverrides(t *testing.T) {
 	assert.False(t, opts.Strict)
 	assert.False(t, opts.NoColors)
 	assert.Equal(t, int64(2), opts.Randomize)
+	assert.Equal(t, 3, opts.MaxRetries)


Check here also. This might need updating. Although given this is the same as the above maybe this is just asserting that?

Same as above, this is to make sure the MaxRetries option can be assigned via CLI.

luke-hill · 2025-03-31T14:35:20Z

internal/flags/options.go

@@ -80,6 +80,9 @@ type Options struct {

 	// ShowHelp enables suite to show CLI flags usage help and exit.
 	ShowHelp bool
+
+	// MaxRetries is used to retry the number of time when a step is failed. Default is 0 retry.


Suggested change

// MaxRetries is used to retry the number of time when a step is failed. Default is 0 retry.

// MaxRetries is the number of times you can retry failing tests. Default is 0.

luke-hill · 2025-03-31T14:37:03Z

retry_flaky_test.go

+
+			fifthTimePass := 0
+			ctx.Step(`^a step that always fails`, func(ctx context.Context) (context.Context, error) {
+				fifthTimePass++


This looks like duplicate info from above. Is there a way to avoid defining this twice.

Oh I forgot, this file is unit tests for the framework itself. So this case here is to cover the case step retried all attempts but still fail.

nhv96 · 2025-04-12T13:44:18Z

Some minor fixes suggested. However a larger question is, are you seeking conformance with standard cucumber execution / cck conformance (I know you might not be there yet).

The reason why, is that a lot of your implementation details surrounding the specific status is unique and not what all standard implementors do

@luke-hill I agree that this implementation works for the specific status godog.ErrRetry, in our discussion thread we also proposed other approaches as well such as using tags @flaky in the feature files.

Do you mean that other standard implementors will also implement such approaches as well? If it is the case, then this PR is to implement the approach using RetryErr, then the following PRs can be to add other approaches.

So I do want to conform the cucumber standard for implementations, but I don't want to make so many different changes in one PR 😄

nhv96 added 2 commits March 29, 2025 20:49

nhv96 mentioned this pull request Mar 29, 2025

Retry failed tests automatically #658

Open

luke-hill requested changes Mar 31, 2025

View reviewed changes

nhv96 added 2 commits April 13, 2025 21:31

make example less obfuscating

5ce208b

update text message

fd83073

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement retry flaky steps #684

Implement retry flaky steps #684

nhv96 commented Mar 29, 2025 •

edited

Loading

luke-hill left a comment

luke-hill Mar 31, 2025

nhv96 Apr 12, 2025

luke-hill Mar 31, 2025

nhv96 Apr 12, 2025

luke-hill Mar 31, 2025

nhv96 Apr 12, 2025

luke-hill Mar 31, 2025

nhv96 Apr 12, 2025

luke-hill Mar 31, 2025

nhv96 Apr 12, 2025

luke-hill Mar 31, 2025

luke-hill Mar 31, 2025

nhv96 Apr 12, 2025

nhv96 commented Apr 12, 2025

	flagSet.IntVar(&opts.MaxRetries, prefix+"retry", opts.MaxRetries, "retry n times when a step encounter error")
	flagSet.IntVar(&opts.MaxRetries, prefix+"retry", opts.MaxRetries, "Specify the number of times to retry failing tests (default: 0)")

	// MaxRetries is used to retry the number of time when a step is failed. Default is 0 retry.
	// MaxRetries is the number of times you can retry failing tests. Default is 0.

Implement retry flaky steps #684

Are you sure you want to change the base?

Implement retry flaky steps #684

Conversation

nhv96 commented Mar 29, 2025 • edited Loading

🤔 What's changed?

⚡️ What's your motivation?

🏷️ What kind of change is this?

♻️ Anything particular you want feedback on?

📋 Checklist:

luke-hill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nhv96 commented Apr 12, 2025

nhv96 commented Mar 29, 2025 •

edited

Loading