Skip to content

Add mix test --dry-run flag #14499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

Nezteb
Copy link
Contributor

@Nezteb Nezteb commented May 15, 2025

Aims to address #14393

@Nezteb Nezteb force-pushed the mix-test-dry-run-flag branch from 0eadca6 to 64bb5e1 Compare May 15, 2025 16:34
Copy link
Member

@whatyouhide whatyouhide left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, implementation-wise I think we might need to take a different approach. Right now, I see two main issues:

  1. We're not running ExUnit.Formatters on dry runs—I think we might want to. Basically, the only thing we want --dry-run to do is to literally skip executing the body of tests (+ setup/setup_all calls and whatnot), but everything else would be the same.
  2. We are printing test files, but this doesn't give us information about what tests (or test cases even, if there are >1 test cases in a file) would run. I think this is important. Think of using --dry-run to answer the question "if I pass this --exclude option, how many tests will be skipped"?.

Do these points make sense? I am not even sure this is very feasible, so we'd need to sort of proof-of-concept this.

@Nezteb Nezteb requested a review from whatyouhide May 16, 2025 16:59
@Nezteb Nezteb force-pushed the mix-test-dry-run-flag branch from d23bce5 to 56e03ed Compare June 2, 2025 23:42
@Nezteb Nezteb requested a review from josevalim June 2, 2025 23:44
@Nezteb
Copy link
Contributor Author

Nezteb commented Jun 17, 2025

@whatyouhide Hello again! Sorry for the ping; I just wanted to check in occasionally so I don't lose track of this PR; I'm in no actual rush. 😄

@whatyouhide
Copy link
Member

@Nezteb I won't be able to review today, but I might find time tomorrow. I put it on my todo list!

Copy link
Member

@whatyouhide whatyouhide left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this locally and there are some issues.

The biggest one is that it prints Test dry run: for files that then exeute no tests. An example from stream_data:

→ mix test --dry-run
Running ExUnit with seed: 188086, max_cases: 28
Excluding tags: [:stdlib]

Test dry run:
Test dry run:
test/stream_data/lazy_tree_test.exs:16
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:77
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:8
test/stream_data/lazy_tree_test.exs:68
test/stream_data/lazy_tree_test.exs:52
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:35
Test dry run:
Test dry run:
Test dry run:
Test dry run:

I think we should print the Test dry run header only once and then just print all tests. Then, we should likely do some formatting. I’m thinking to use color in the header and call it something like Tests that would be executed:, in :cyan (respecting color settings for Mix).

Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants