Reduce flake in Fleet installer tests (#6671)

## Summary of changes

Reduce the flake seen in Fleet installer tests

## Reason for change

The fleet installer tests are currently very flaky. This is mostly due
to the nightmare of trying to get the app to behave as it does elsewhere
(which _maybe_ we should just give up on, more on that later).

## Implementation details

The simple details are:
- Add a delay after running `ServiceMonitor.exe` before starting the app
pool and sending the initial request
- Add a hacky _completed.txt_ file to allow tracking when the app has
shutdown.

First, some history...

The design of the test app is that it starts, wait for itself to be
ready to handle requests, sends a self-request, and then shuts down.
This works great for Docker outside of IIS, because it means we know
that we get exactly one request, once the app is ready, and the docker
container exits once the sequence is complete. :chef-kiss:

On IIS, this just doesn't work. IIS does its best to make sure the app
_is_ always running. There's a bunch of other complexity. One of which
is the only way to get the app running in the first place is to send a
request to it, which messes with our snapshots (that's already handled).
But a bigger issue is the shutdown - I just can't find a way to "wait"
for it using the built-in behaviour. And without being able to wait for
shutdown, we don't know exactly when the "expected" request has been
sent.

So this has a hack - write to a file in the app when the app is shutting
down. And spin in the powershell script, waiting for that file to exist.

## Test coverage

I've run the tests repeatedly on this branch, and not seen any failures,
so 🤞

## Other details

Yeah, I know, the arbitrary delay could be a source of flake. I spent
_way_ too long trying to capture the output to explicitly wait for the
expected output, but no matter what sort of redirection I tried, I
couldn't capture it as I needed - it only writes to the redirected
output when the process ends, and the process _doesn't_ end 🙄 So yeah,
this is a hack, but it works

Author: andrewlock

tracer/build/_build/docker/smoke.windows.fleet-installer.dockerfile

@@ -51,21 +51,24 @@ ENV DD_INTERNAL_WORKAROUND_77973_ENABLED=1
 # Copy the app across
 COPY --from=builder /src/publish /app/.
 
-# Create new website we control, but keep auto-start disabled
+# Create new website we control, but keep auto-start disabled, and stop IIS
 RUN Remove-WebSite -Name 'Default Web Site'; \
     $ENABLE_32_BIT='false'; \
     if($env:TARGET_PLATFORM -eq 'x86') { $ENABLE_32_BIT='true' }; \
     Write-Host "Creating website with 32 bit enabled: $env:ENABLE_32_BIT"; \
     c:\Windows\System32\inetsrv\appcmd add apppool /startMode:"AlwaysRunning" /autoStart:"false" /name:AspNetCorePool /managedRuntimeVersion:"" /enable32bitapponwin64:$ENABLE_32_BIT; \
     New-Website -Name 'SmokeTest' -Port 5000 -PhysicalPath 'c:\app' -ApplicationPool 'AspNetCorePool'; \
-    Set-ItemProperty "IIS:\Sites\SmokeTest" -Name applicationDefaults.preloadEnabled -Value True;
+    Set-ItemProperty "IIS:\Sites\SmokeTest" -Name applicationDefaults.preloadEnabled -Value True; \
+    net stop /y was
 
 # We override the normal service monitor entrypoint, because we want the container to shut down after the request is sent
-# We would really like to get the pid of the worker processes, but we can't do that easily
 # This is all way more convoluted than it feels like it should be, but it's the only way I could find to get things to work as required
-RUN echo 'Write-Host \"Running servicemonitor to copy variables\"; Start-Process -NoNewWindow -PassThru -FilePath \"c:/ServiceMonitor.exe\" -ArgumentList @(\"w3svc\", \"AspNetCorePool\");' > C:\app\entrypoint.ps1; \
+RUN echo '$completedFile=\"C:\logs\completed.txt\"; if (Test-Path $completedFile) { Remove-Item $completedFile;};'  > C:\app\entrypoint.ps1; \
+    echo 'Write-Host \"Running servicemonitor to copy variables\"; Start-Process -NoNewWindow -PassThru -FilePath \"c:/ServiceMonitor.exe\" -ArgumentList @(\"w3svc\", \"AspNetCorePool\");' >> C:\app\entrypoint.ps1; \
+    echo 'Write-Host \"Waiting 10s before starting app pool\"; Start-Sleep -Seconds 10;' >> C:\app\entrypoint.ps1; \
     echo 'Write-Host \"Starting AspNetCorePool app pool\"; Start-WebAppPool -Name \"AspNetCorePool\" -PassThru;' >> C:\app\entrypoint.ps1; \
     echo 'Write-Host \"Making 404 request\"; curl http://localhost:5000;' >> C:\app\entrypoint.ps1; \
+    echo 'while (-not (Test-Path "C:\logs\completed.txt")) { Write-Host \"Waiting for app shutdown\"; Start-Sleep -Seconds 1; }; ' >> C:\app\entrypoint.ps1; \
     echo 'Write-Host \"Stopping pool\";Stop-WebAppPool \"AspNetCorePool\" -PassThru;' >> C:\app\entrypoint.ps1;  \
     echo 'Write-Host \"Shutting down\"' >> C:\app\entrypoint.ps1;
 

tracer/test/test-applications/regression/AspNetCoreSmokeTest/Program.cs

@@ -1,6 +1,7 @@
 using Microsoft.AspNetCore.Hosting;
 using Microsoft.Extensions.Hosting;
 using System;
+using System.IO;
 using System.Net.Http;
 using System.Reflection;
 using System.Runtime.InteropServices;
@@ -36,6 +37,12 @@ public static async Task<int> Main(string[] args)
 
             await CreateHostBuilder(args).Build().RunAsync();
 
+            var completedPath = Path.Combine(
+                Environment.GetEnvironmentVariable("DD_TRACE_LOG_DIRECTORY") ?? ".",
+                "completed.txt");
+
+            File.WriteAllText(completedPath, DateTimeOffset.UtcNow.ToString());
+
             return ExitCode;
         }