Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder has flaky test assertion

[ericstj]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1056990
Build error leg or test failing: dotnet-watch.Tests.dll.3.WorkItemExecution
Pull request: #49092

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Failed Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Sample failing log: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-49092-merge-89284afdacb94b978c/dotnet-watch.Tests.dll.3/3/console.3de535af.log?helixlogtype=result

  Failed Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder(usePolling: True) [1 s]
  Error Message:
   Expected:
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo1, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo2, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo3, Kind = Delete }
Actual:
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir, Kind = Update },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo1, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo2, Kind = Delete }

  Stack Trace:
     at Microsoft.DotNet.Watch.UnitTests.AssertEx.Fail(String message) in /_/test/dotnet-watch.Tests/Utilities/AssertEx.cs:line 214
   at Microsoft.DotNet.Watch.UnitTests.AssertEx.SequenceEqual[T](IEnumerable`1 expected, IEnumerable`1 actual, IEqualityComparer`1 comparer, String message, String itemSeparator, Func`2 itemInspector) in /_/test/dotnet-watch.Tests/Utilities/AssertEx.cs:line 169
   at Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.TestOperation(String dir, ChangedPath[] expectedChanges, Boolean usePolling, Action operation) in /_/test/dotnet-watch.Tests/FileWatcherTests.cs:line 63
   at Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder(Boolean usePolling) in /_/test/dotnet-watch.Tests/FileWatcherTests.cs:line 353
--- End of stack trace from previous location ---
  Standard Output Messages:

Have a look at the assertion the test is making --

sdk/test/dotnet-watch.Tests/FileWatcherTests.cs

Lines 353 to 390 in 0934942

	await TestOperation(
	dir,
	expectedChanges: usePolling ?
	[
	new(subdir, ChangeKind.Delete),
	new(f1, ChangeKind.Delete),
	new(f2, ChangeKind.Delete),
	new(f3, ChangeKind.Delete),
	]
	: RuntimeInformation.IsOSPlatform(OSPlatform.OSX) ?
	[
	new(subdir, ChangeKind.Add),
	new(subdir, ChangeKind.Delete),
	new(f1, ChangeKind.Update),
	new(f1, ChangeKind.Add),
	new(f1, ChangeKind.Delete),
	new(f2, ChangeKind.Update),
	new(f2, ChangeKind.Add),
	new(f2, ChangeKind.Delete),
	new(f3, ChangeKind.Update),
	new(f3, ChangeKind.Add),
	new(f3, ChangeKind.Delete),
	]
	: RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ?
	[
	new(subdir, ChangeKind.Update),
	new(subdir, ChangeKind.Delete),
	new(f1, ChangeKind.Delete),
	new(f2, ChangeKind.Delete),
	new(f3, ChangeKind.Delete),
	]
	:
	[
	new(subdir, ChangeKind.Delete),
	new(f1, ChangeKind.Delete),
	new(f2, ChangeKind.Delete),
	new(f3, ChangeKind.Delete),
	],

It's asserting it will see a series of events exactly on different operating systems. FileSystemWatcher (and file notification APIs in the platform for that matter) don't garuntee that. You can't count on seeing all changed/updated events. Especially in the case of polling you cannot garuntee you'll see an update event if it's followed by a delete since a polling implementation can only see a file or directory has been removed, and not see any changes that might have happened between the last poll and removal.

This test should be changed to just assert on seeing a minimum set of events, not exact set. In other words - a contains check rather than equals.

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1056990
Error message validated: [Failed Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/2/2025 8:08:17 PM UTC

Report

Build	Definition	Test	Pull Request
1067970	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49395
1065689	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49329
1066049	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49309
1065990	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#48905
1065782	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49288
1064585	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49306
1064372	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#49336
1063370	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#48387
1063334	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49287
1062853	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49037
1061971	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49295
1061470	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49290
2724657	dotnet-sdk	dotnet-watch.Tests.dll.3.WorkItemExecution
1060349	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#48575
1059964	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49241
1058657	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49241
1056990	dotnet/sdk	dotnet-watch.Tests.dll.3.WorkItemExecution	#49092

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	7	17