input: add new histogram metric for size of records

[nuclearpidgeon]

Currently it's not easy to reason about the size of records passing through Fluentbit over time - e.g. catching if a lot of large records start coming through. You can get a loose proxy by dividing fluentbit_input_bytes_total by fluentbit_intput_records_total, but this is a complete average. Histograms can provide more granular detail in cases like this. This PR adds a new histogram metric fluentbit_input_record_sizes that observes the size of each input record, alongside the existing total input bytes metric.

The default bucket sizes are just a stab in the dark. It would probably be ideal for them to be configurable, but this will involve more work and e.g. limit checks.

Testing

Fluentbit command for testing some input data, with the metrics server enabled
fluent-bit -i tail -p path=./logslurp -p buffer_max_size=4m -o file -m '*' -H
Pump some random data into the input
for size in '100B' '1K' '2K' '4K' '1M';do echo $size; cat /dev/random | base64 -w 0 | dd of=./logslurp bs=$size count=1 oflag=append conv=notrunc;printf "\n" >> logslurp; done
Hit the metrics endpoint to see the metrics:
curl -v localhost:2020/api/v2/metrics/prometheus | grep sizes:

# HELP fluentbit_input_record_sizes Histogram of the size of input records
# TYPE fluentbit_input_record_sizes histogram
fluentbit_input_record_sizes_bucket{le="0.0",name="tail.0"} 0
fluentbit_input_record_sizes_bucket{le="100.0",name="tail.0"} 0
fluentbit_input_record_sizes_bucket{le="1024.0",name="tail.0"} 1
fluentbit_input_record_sizes_bucket{le="2048.0",name="tail.0"} 2
fluentbit_input_record_sizes_bucket{le="4096.0",name="tail.0"} 3
fluentbit_input_record_sizes_bucket{le="102400.0",name="tail.0"} 5
fluentbit_input_record_sizes_bucket{le="1.04858e+06",name="tail.0"} 5
fluentbit_input_record_sizes_bucket{le="4.1943e+06",name="tail.0"} 5
fluentbit_input_record_sizes_bucket{le="+Inf",name="tail.0"} 0
fluentbit_input_record_sizes_sum{name="tail.0"} 48412
fluentbit_input_record_sizes_count{name="tail.0"} 5

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.