Add metric that monitors fallback candidate health

[dpulitano]

Description

It is useful to be able to alert when all of the fallback candidates are unhealthy or if the healthy fallback candidates have fallen below a certain level.

This PR adds a gauge to monitor this whenever the primary candidates are all unhealthy

Note that this also adds consensus polling for fallback candidates when there are healthy primary backends available (earlier, consensus polling was only performed if no primary candidates were healthy). I acknowledge that maybe this behavior might be undesired for some people...
I'm open to creating a CLI flag to disable this behavior, but personally think that the default should be to monitor for fallback candidate health so that the metric is by default populated. Open to suggestions about this from core maintainers.

Tests

Testing in-progress, but wanted to publish the PR with the note about consensus polling for fallbacks behavior

Additional context

Add any other context about the problem you're solving.

Metadata

• Fixes #[Link to Issue]

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.28%. Comparing base (7d490c1) to head (c641ffc).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #417      +/-   ##
==========================================
- Coverage   61.32%   61.28%   -0.04%     
==========================================
  Files          80       80              
  Lines       10855    10860       +5     
==========================================
- Hits         6657     6656       -1     
- Misses       3780     3785       +5     
- Partials      418      419       +1     

Flag	Coverage Δ
proxyd	71.58% <100.00%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
proxyd/consensus_poller.go	80.95% <100.00%> (+0.10%)	⬆️
proxyd/metrics.go	84.31% <100.00%> (+0.31%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.