-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Add mix test --dry-run flag #14499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Add mix test --dry-run flag #14499
Conversation
0eadca6
to
64bb5e1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, implementation-wise I think we might need to take a different approach. Right now, I see two main issues:
- We're not running
ExUnit.Formatter
s on dry runs—I think we might want to. Basically, the only thing we want--dry-run
to do is to literally skip executing the body of tests (+setup
/setup_all
calls and whatnot), but everything else would be the same. - We are printing test files, but this doesn't give us information about what tests (or test cases even, if there are >1 test cases in a file) would run. I think this is important. Think of using
--dry-run
to answer the question "if I pass this--exclude
option, how many tests will be skipped"?.
Do these points make sense? I am not even sure this is very feasible, so we'd need to sort of proof-of-concept this.
d23bce5
to
56e03ed
Compare
@whatyouhide Hello again! Sorry for the ping; I just wanted to check in occasionally so I don't lose track of this PR; I'm in no actual rush. 😄 |
@Nezteb I won't be able to review today, but I might find time tomorrow. I put it on my todo list! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried this locally and there are some issues.
The biggest one is that it prints Test dry run:
for files that then exeute no tests. An example from stream_data:
→ mix test --dry-run
Running ExUnit with seed: 188086, max_cases: 28
Excluding tags: [:stdlib]
Test dry run:
Test dry run:
test/stream_data/lazy_tree_test.exs:16
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:77
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:8
test/stream_data/lazy_tree_test.exs:68
test/stream_data/lazy_tree_test.exs:52
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:6
test/stream_data/lazy_tree_test.exs:35
Test dry run:
Test dry run:
Test dry run:
Test dry run:
I think we should print the Test dry run
header only once and then just print all tests. Then, we should likely do some formatting. I’m thinking to use color in the header and call it something like Tests that would be executed:
, in :cyan
(respecting color settings for Mix).
Thoughts?
Aims to address #14393