Fixing flaky test (#780)

crawlingcub · facebook-github-bot · commit 9b33f13aabdb · 2021-10-13T08:51:48.000-07:00
Summary: Running some tests by removing seeds revealed that tests such as `test_single_shapley_int_batch_scalar_float`, `test_single_shapley_int_batch_scalar_tensor_0d`, `test_single_shapley_int_batch_scalar_tensor_1d`, and `test_single_shapley_int_batch_scalar_tensor_int` are flaky. They almost fail 40-50% of the times. To fix these tests, I updated the `n_samples` to 2500 to fix the flakiness. The tests now pass consistently. This only affects the execution time of the test by ~1s. Please let me know if this change seems reasonable. Similar to #775. Pull Request resolved: #780 Reviewed By: NarineK, Reubend Differential Revision: D31587778 Pulled By: vivekmig fbshipit-source-id: 843adda41662e881a6bf81ebc4354d6f304e83ce
diff --git a/tests/attr/test_shapley.py b/tests/attr/test_shapley.py
@@ -349,6 +349,7 @@ def _single_int_input_multi_sample_batch_scalar_shapley_assert(
             feature_mask=mask,
             perturbations_per_eval=(1,),
             target=None,
+            n_samples=2500,
         )
 
     def _multi_input_batch_scalar_shapley_assert(self, func: Callable) -> None:

Original file line number	Diff line number	Diff line change
`@@ -349,6 +349,7 @@ def _single_int_input_multi_sample_batch_scalar_shapley_assert(`
`349`	`349`	`feature_mask=mask,`
`350`	`350`	`perturbations_per_eval=(1,),`
`351`	`351`	`target=None,`
	`352`	`+ n_samples=2500,`
`352`	`353`	`)`
`353`	`354`
`354`	`355`	`def _multi_input_batch_scalar_shapley_assert(self, func: Callable) -> None:`