Add support for timestamp encoding in exemplars #276

bobrik · 2025-08-07T21:48:31Z

While timestamps are optional, they are required for native histograms:

https://github.com/prometheus/prometheus/blame/4aee718013/scrape/scrape.go#L1833

Old histograms can be converted to native histograms at the ingestion time, which in turn makes timestamps required for old histograms too.

bobrik · 2025-08-07T22:16:18Z

Clippy complaints are from Rust v1.89 (they should probably be addressed separately):

https://blog.rust-lang.org/2025/08/07/Rust-1.89.0/#mismatched-lifetime-syntaxes-lint

mxinden

Thanks for the help.

In favor. Just one discussion point.

Missing changelog entry.

Agreed with clippy fixes in a separate pull request? Would you mind helping out here?

mxinden · 2025-08-08T08:12:38Z

src/metrics/exemplar.rs

        inner.exemplar = label_set.map(|label_set| Exemplar {
            label_set,
            value: v.clone(),
+            time: SystemTime::now(),


How expensive is SystemTime::now?

My intuition is, that it is orders of magnitude larger than a simple Prometheus Counter atomic increase.

If so, is that performance impact intuitive for users? Is it worth it for users not using exemplar timestamps?

If not, what do you think of a mechanism that makes these optional?

Let me do some benchmarking to have some numbers.

We can hide this behind a feature if it's too expensive, if you're okay with that approach.

I added benchmarks and the following to the commit message. Let me know if you would prefer to have this to be feature gated still.

Benchmarking this against the baseline with no changes:

AMD EPYC 7642:

histogram without exemplars time: [12.389 ns 12.390 ns 12.392 ns] change: [-0.0820% -0.0532% -0.0227%] (p = 0.00 < 0.05) Change within noise threshold. histogram with exemplars (no exemplar passed) time: [28.469 ns 28.476 ns 28.483 ns] change: [+1.9145% +1.9533% +1.9954%] (p = 0.00 < 0.05) Performance has regressed. histogram with exemplars (some exemplar passed) time: [135.70 ns 135.83 ns 135.96 ns] change: [+49.325% +49.740% +50.112%] (p = 0.00 < 0.05) Performance has regressed.

Apple M3 Pro:

histogram without exemplars time: [3.1357 ns 3.1617 ns 3.1974 ns] change: [+1.2045% +2.0927% +3.1167%] (p = 0.00 < 0.05) Performance has regressed. histogram with exemplars (no exemplar passed) time: [5.8648 ns 5.8751 ns 5.8872 ns] change: [+0.1479% +0.9875% +1.6873%] (p = 0.01 < 0.05) Change within noise threshold. histogram with exemplars (some exemplar passed) time: [69.448 ns 69.790 ns 70.192 ns] change: [+24.346% +24.897% +25.459%] (p = 0.00 < 0.05) Performance has regressed.

The only real change would come in the third benchmark when exemplars are actually passed, changes in the other two are due to noise.

Exemplars are usually used for tracing, where there's sampling involved, so not every observation would incur a performance penalty, and only a small fraction would be affected. For cases where tracing is enabled for 100% of observations, the overhead if tracing itself would drown any changes here.

bobrik

Missing changelog entry.

I'll add it.

Agreed with clippy fixes in a separate pull request? Would you mind helping out here?

See #277

bobrik · 2025-08-08T18:00:13Z

src/metrics/exemplar.rs

        inner.exemplar = label_set.map(|label_set| Exemplar {
            label_set,
            value: v.clone(),
+            time: SystemTime::now(),


Let me do some benchmarking to have some numbers.

We can hide this behind a feature if it's too expensive, if you're okay with that approach.

mxinden

🙏 thanks for the benchmark. Very helpful.

As an aside, I am surprised that histogram with exemplars (no exemplar passed) doesn't equal histogram without exemplars. Is the formatting different?

Instead of a feature flag, how about changing the CounterWithExemplar and HistogramWithExemplar methods slightly?

-     pub fn observe(&self, v: f64, label_set: Option<S>) {
+     pub fn observe(&self, v: f64, label_set: Option<S>, timestamp: Option<SystemTime>) {

Benefits:

Users that don't need the timestamp don't pay for it. Given that it is explicit in the signature, there are no surprises.
Users that already have a current SystemTime around, can re-use it.

Thoughts?

bobrik · 2025-08-11T23:38:20Z

As an aside, I am surprised that histogram with exemplars (no exemplar passed) doesn't equal histogram without exemplars. Is the formatting different?

When you observe a measurement for Histogram, you take one lock. With HistogramWithExemplar there's inner Histogram locking plus an outer lock to protect HistogramWithExemplarsInner as mentioned by you:

client_rust/src/metrics/exemplar.rs

Lines 214 to 215 in 1d5824a

    
           // TODO: Not ideal, as Histogram has a Mutex as well. 
        
           pub(crate) inner: Arc<RwLock<HistogramWithExemplarsInner<S>>>,

You can separate exemplar locking from the histogram locking, but it doesn't become much faster. Interestingly, it speeds up quite a bit if you insert a false panic that never triggers:

    pub fn observe(&self, v: f64, label_set: Option<S>) {
        let bucket = self.inner.histogram.observe_and_bucket(v);
        if let (Some(bucket), Some(label_set)) = (bucket, label_set) {
            // if true {
            //     panic!("wtf");
            // }
            self.inner.exemplars.write().insert(
                bucket,
                Exemplar {
                    label_set,
                    value: v,
                    time: SystemTime::now(),
                },
            );
            // self.observe_exemplar(bucket, v, label_set);
        }
    }

histogram with exemplars (no exemplar passed)
                        time:   [3.6716 ns 3.6808 ns 3.6916 ns]
                        change: [-30.420% -29.909% -29.374%] (p = 0.00 < 0.05)
                        Performance has improved.

One needs to look at assembly to figure out what drives this difference. That's on Apple M3 Pro.

On AMD it is faster even without the fake panic and the panic doesn't move the needle further:

histogram with exemplars (no exemplar passed)
                        time:   [15.697 ns 15.706 ns 15.716 ns]
                        change: [-45.275% -45.202% -45.129%] (p = 0.00 < 0.05)
                        Performance has improved.

It's definitely a candidate for another PR.

Instead of a feature flag, how about changing the CounterWithExemplar and HistogramWithExemplar methods slightly?

It's certainly an option, but given that it's the public interface it would be a breaking change. Adding timestamps implicitly avoids that.

Allowing to pass a timestamp is nice when you have multiple histograms to update around the same time.

There is some cost to it:

histogram without exemplars
                        time:   [3.1824 ns 3.2017 ns 3.2235 ns]
                        change: [-3.6414% +0.1893% +3.1022%] (p = 0.92 > 0.05)
                        No change in performance detected.

histogram with exemplars (no exemplar passed)
                        time:   [6.2238 ns 6.2487 ns 6.2770 ns]
                        change: [+7.5129% +8.3245% +9.0265%] (p = 0.00 < 0.05)
                        Performance has regressed.

histogram with exemplars (some exemplar passed)
                        time:   [70.907 ns 71.667 ns 72.493 ns]
                        change: [+1.4206% +2.3509% +3.3463%] (p = 0.00 < 0.05)
                        Performance has regressed.

I updated the code.

mxinden · 2025-08-14T07:42:51Z

src/metrics/exemplar.rs

 pub struct Exemplar<S, V> {
    pub(crate) label_set: S,
    pub(crate) value: V,
+    pub(crate) time: SystemTime,


Sorry for the confusion. I suggest the following:

Suggested change

pub(crate) time: SystemTime,

pub(crate) time: Option<SystemTime>,

Then in CounterWithExemplar::inc_by:

pub fn inc_by(&self, v: N, label_set: Option<S>, timestamp: Option<SystemTime>) -> N {

And HistogramWithExemplar:

pub fn observe(&self, v: f64, label_set: Option<S>, timestamp: Option<SystemTime>) { let mut inner = self.inner.write(); let bucket = inner.histogram.observe_and_bucket(v); if let (Some(bucket), Some(label_set)) = (bucket, label_set) { inner.exemplars.insert( bucket, Exemplar { label_set, value: v, - time: timestamp.unwrap_or_else(SystemTime::now), + time: timestamp, }, ); }

That way users that need a timestamp can easily enable it, while users that don't don't pay for it.

Am I missing something?

Regarding breaking change, I think that is fine. Next version is a breaking change anyways. I doubt many folks use the exemplar API. I think the signature change is self-explanatory.

Sounds good, I updated the code.

mxinden · 2025-08-14T07:43:35Z

CHANGELOG.md

+- Exemplar timestamps, which are required for `convert_classic_histograms_to_nhcb: true`
+  in Prometheus scraping. See [PR 276].


In case we decide for the breaking change, better included under ### Changed.

I moved it there.

mxinden · 2025-08-14T07:44:28Z

As an aside, I am surprised that histogram with exemplars (no exemplar passed) doesn't equal histogram without exemplars. Is the formatting different?

When you observe a measurement for Histogram, you take one lock. With HistogramWithExemplar there's inner Histogram locking plus an outer lock to protect HistogramWithExemplarsInner as mentioned by you:

🙏

Signed-off-by: Ivan Babrou <[email protected]>

While timestamps are optional, they are required for native histograms: * https://github.com/prometheus/prometheus/blame/4aee718013/scrape/scrape.go#L1833 Old histograms can be converted to native histograms at the ingestion time, which in turn makes timestamps required for old histograms too. Benchmarking this against the baseline with no changes: * AMD EPYC 7642: ``` histogram without exemplars time: [12.389 ns 12.390 ns 12.392 ns] change: [-0.0820% -0.0532% -0.0227%] (p = 0.00 < 0.05) Change within noise threshold. histogram with exemplars (no exemplar passed) time: [28.469 ns 28.476 ns 28.483 ns] change: [+1.9145% +1.9533% +1.9954%] (p = 0.00 < 0.05) Performance has regressed. histogram with exemplars (some exemplar passed) time: [135.70 ns 135.83 ns 135.96 ns] change: [+49.325% +49.740% +50.112%] (p = 0.00 < 0.05) Performance has regressed. ``` * Apple M3 Pro: ``` histogram without exemplars time: [3.1357 ns 3.1617 ns 3.1974 ns] change: [+1.2045% +2.0927% +3.1167%] (p = 0.00 < 0.05) Performance has regressed. histogram with exemplars (no exemplar passed) time: [5.8648 ns 5.8751 ns 5.8872 ns] change: [+0.1479% +0.9875% +1.6873%] (p = 0.01 < 0.05) Change within noise threshold. histogram with exemplars (some exemplar passed) time: [69.448 ns 69.790 ns 70.192 ns] change: [+24.346% +24.897% +25.459%] (p = 0.00 < 0.05) Performance has regressed. ``` The only real change would come in the third benchmark when exemplars are actually passed, changes in the other two are due to noise. Exemplars are usually used for tracing, where there's sampling involved, so not every observation would incur a performance penalty, and only a small fraction would be affected. For cases where tracing is enabled for 100% of observations, the overhead if tracing itself would drown any changes here. Signed-off-by: Ivan Babrou <[email protected]>

This allows having exemplars without timestamps, which are cheaper. It also allows amortization of timestamping if multiple measurements share time. Signed-off-by: Ivan Babrou <[email protected]>

mxinden

Thanks for bearing with me!

bobrik · 2025-08-14T20:33:13Z

@mxinden, when do you expect to have a release?

Signed-off-by: Ivan Babrou <[email protected]> Signed-off-by: Max Leonard Inden <[email protected]>

mxinden · 2025-08-15T12:26:32Z

@bobrik done:

bobrik force-pushed the ivan/exemplar-timestamps branch from 52866fd to 0486185 Compare August 7, 2025 22:07

mxinden reviewed Aug 8, 2025

View reviewed changes

bobrik commented Aug 8, 2025

View reviewed changes

bobrik force-pushed the ivan/exemplar-timestamps branch from 0486185 to a769bf5 Compare August 8, 2025 21:54

mxinden reviewed Aug 11, 2025

View reviewed changes

bobrik force-pushed the ivan/exemplar-timestamps branch from a769bf5 to 7951bc8 Compare August 11, 2025 23:38

bobrik force-pushed the ivan/exemplar-timestamps branch 3 times, most recently from 3937878 to 0e1bb0e Compare August 12, 2025 04:11

mxinden reviewed Aug 14, 2025

View reviewed changes

bobrik force-pushed the ivan/exemplar-timestamps branch 2 times, most recently from 920562d to 3a9225b Compare August 14, 2025 16:15

bobrik added 2 commits August 14, 2025 09:16

Add a benchmark for exemplars

14e18ba

Signed-off-by: Ivan Babrou <[email protected]>

bobrik force-pushed the ivan/exemplar-timestamps branch from 3a9225b to b698bfa Compare August 14, 2025 16:18

Pass exemplar timestamp explicitly

9435495

This allows having exemplars without timestamps, which are cheaper. It also allows amortization of timestamping if multiple measurements share time. Signed-off-by: Ivan Babrou <[email protected]>

bobrik force-pushed the ivan/exemplar-timestamps branch from b698bfa to 9435495 Compare August 14, 2025 16:22

mxinden approved these changes Aug 14, 2025

View reviewed changes

mxinden merged commit 0355911 into prometheus:master Aug 14, 2025
10 checks passed

mxinden pushed a commit to mxinden/client_rust that referenced this pull request Aug 15, 2025

feat: support timestamp encoding in exemplars (prometheus#276)

529a128

Signed-off-by: Ivan Babrou <[email protected]> Signed-off-by: Max Leonard Inden <[email protected]>

	pub(crate) time: SystemTime,
	pub(crate) time: Option<SystemTime>,

		- Exemplar timestamps, which are required for `convert_classic_histograms_to_nhcb: true`
		in Prometheus scraping. See [PR 276].

Add support for timestamp encoding in exemplars #276

Add support for timestamp encoding in exemplars #276

Uh oh!

Conversation

bobrik commented Aug 7, 2025

Uh oh!

bobrik commented Aug 7, 2025

Uh oh!

mxinden left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bobrik left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mxinden left a comment

Choose a reason for hiding this comment

Uh oh!

bobrik commented Aug 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mxinden commented Aug 14, 2025

Uh oh!

mxinden left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bobrik commented Aug 14, 2025

Uh oh!

mxinden commented Aug 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants