Skip to content

Commit b6f6286

Browse files
crawlingcubfacebook-github-bot
authored andcommitted
Fixing flaky tests (#781)
Summary: This is a fix for another set of flaky tests: `test_multi_inp_shapley_batch_scalar_tensor_0d`, `test_multi_inp_shapley_batch_scalar_float` and `test_multi_inp_shapley_batch_scalar_tensor_1d` in `test_shapley.py`. Note that `test_multi_inp_shapley_batch_scalar_int` was not flaky to begin with. I observed that increasing `n_samples` up to 3500 reduces flakiness by a lot. This increases the runtime from about 1.7s to 4.5s though. The tests only failed about 6/7 times out of 1000 runs with only the `n_samples` change. Hence, I also increased the `delta` to 1.2 based on the observed failures to bring that to ~0%. Let me know in case you would prefer lower `n_samples` but higher `delta` to reduce the runtime. Thanks! cc vivekmig Pull Request resolved: #781 Reviewed By: NarineK, Reubend Differential Revision: D31588284 Pulled By: vivekmig fbshipit-source-id: d398ad5b54629f6e472309e9b66352b2e8454691
1 parent 6a1e261 commit b6f6286

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

tests/attr/test_shapley.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -372,7 +372,8 @@ def _multi_input_batch_scalar_shapley_assert(self, func: Callable) -> None:
372372
feature_mask=(mask1, mask2, mask3),
373373
perturbations_per_eval=(1,),
374374
target=None,
375-
n_samples=800,
375+
n_samples=3500,
376+
delta=1.2,
376377
)
377378

378379
def _shapley_test_assert(

0 commit comments

Comments
 (0)