Replace applicable `ConcurrentBag` usages with custom lightweight thread-safe collection

[Flash0ver]

Description

In #3431 (fixed by #3432) and #4709 (fixed by #4717),
we noticed that Transaction and Span types utilizing System.Collections.Concurrent.ConcurrentBag may cause memory leaks in our usages.
Most System.Collections.Concurrent collections have additional semantics and features, that we don't really need in most use cases (like IProducerConsumerCollection<T>, as mentioned in #3432 (comment) and #3432 (comment).
In most cases, we just need a thread-safe code path of adding new items to a collection, and later iterating through that collection, in a concurrent fashion.

Requirements
The features / semantics we seem to require are

• Create Empty
• Create with initial Capacity
• Add
• Enumerate
• Clear
• return to user code as IReadOnlyCollection<T>

Some potential ideas

1. use BlockingCollection<T> instead
   • alleviates the memory leak for our Transaction and Span types
   • but has semantics of "Completion" that we don't need in most cases
2. build a simple custom concurrent Collection, backed by a List<T>, that uses a blocking lock for synchronization
   • quick implementation
   • synchronous blocking is not quite ideal
     • especially when trying to add while already being concurrently enumerated
3. build a complex custom concurrent Collection, that is backed by an Array and behaves similar to the double-in-size growing algorithm of a List<T> when adding new items, based on Interlocked.Increment to determine the index to add to
   • non-blocking when non-growing (i.e. adding when the Count is less than the Capacity)
   • blocking when growing (i.e. adding when the Count equals the Capacity)
4. an "Append-Segment" based implementation, similar to how System.Text.StringBuilder works
   • does not discard existing Arrays when growing
   • Memory may not be contiguous, hence Enumeration may be slower

[linear]

NET-521

[ericsampson]

Isn't that what System.Threading.Channels is designed for? It's pretty highly optimized, and has optional flags for SingleProducer and SingleConsumer to further optimize perf for those conditions.
https://devblogs.microsoft.com/dotnet/an-introduction-to-system-threading-channels/