Add resiliency mechanism to reduce exception rate in resource monitoring

[amadeuszl]

Reading files in Linux may throw exceptions, eg. file may not exist or be unaccessible, especially in containerized scenarios. We should consider to build mechanism to skip reading those files if already failed to read it once. We should consider retry too after some time.

Also, it's good verify same mechanisms in Windows.

[amadeuszl]

@evgenyfedorov2 I have a question regarding CPU and memory. I plan to propose following solution:

internal sealed class RetryingLinuxUtilizationParser : ILinuxUtilizationParser
{
    private readonly ILinuxUtilizationParser _inner;
    private T Retry<T>(Func<T> func) // _inner will be passed here
    {
        if (Volatile.Read(ref _unavailable) == 1 && DateTimeOffset.UtcNow - _lastFailure < _retryInterval)
        {
            return; // Could return just 0
        }

        try
        {
            var result = func();
            Interlocked.Exchange(ref _unavailable, 0);
            return result;
        }
        catch (Exception ex) when (
            ex is FileNotFoundException ||
            ex is DirectoryNotFoundException ||
            ex is UnauthorizedAccessException)
        {
            _lastFailure = DateTimeOffset.UtcNow;
            Interlocked.Exchange(ref _unavailable, 1);
            throw; // or return 0
        }
    }
}

But I'm struggling to find out proper way what should be returned when run is skipped, eg. just 0? It would work then the same as in #6489 but it may break some calculations, can verify that. But if it would return 0 then from catch it should return 0 as well. What do you think?

[evgenyfedorov2]

1. can we use some more sophisticated mechanism such as Polly v8 here?
2. can we try to make this Retry method non-blocking and just never return until it succeeds?

[amadeuszl]

1. As it's fairly simple mechanism I want to avoid bringing very big dependencies, I would rather not overcomplicate it.
2. I think approach with returning asap is just fine, I'm not sure polling inside CreateObservableGauge is safe, I'm not exactly sure how it would impact this whole metering pipeline even if non-blocking. Other thing is, I mostly added it for a corner case and convenience if underlying file will become readable at some point. In 99.99% of cases I would assume it won't be readable through the service's lifecycle, so not returning anything never seems even more riskier. But as mentioned I'm not fully aware of ramifications of polling inside CreateObservableGauge