Skip to content

Conversation

@MrWong99
Copy link
Contributor

@MrWong99 MrWong99 commented Sep 5, 2025

Split Loki Push Requests based on labels/structured metadata

This PR adds a new subpath to POST /loki/push that accepts Loki Ingester Push.
It also splits the streams of logs via their labels or structured metadata.

A new configuration option target_loki can be used to configure the Loki endpoint to send the streams to.
The new subpath has its own metrics (named streams instead of timeseries).

I tried to keep the common code within processor.go but needed to split the logic between logs and metrics. The old code can be found in processor_metrics_test.go and handler_metrics.go. The new files for logs are almost identical to the metrics counterpart except for the differing datatypes.

There is also a very minor performance improvement in https://github.com/MrWong99/cortex-tenant/blob/loki/handler_metrics.go#L228 as the old code iterated the outer loop over and over. It will now break on first match but to keep it compatible to the old code I reversed the cfg.Tenant.LabelList after it has been read.

Sorry for the bigger Pull Request...
We faced the same issue with Label -> Header conversion with Grafana Loki.
I looked at the cortex-tenant code and it seemed straightforward to add this fuctionality, so I did.

@blind-oracle
Copy link
Owner

Thanks a lot! It's a big thing, I'll try to find some time to read through :)

@MrWong99
Copy link
Contributor Author

MrWong99 commented Sep 8, 2025

Ah sorry, I am seeing proto parsing errors in our integration tests. I'll fix and push them first. Sorry if you already took a look 😞

@MrWong99
Copy link
Contributor Author

Now with the latest changes our integration tests run smooth. Sorry for the delay...

We also were thinking about adding support for metrics and logs in OpenTelemetry format, but this will be another PR if we decide to support this 😄

@blind-oracle blind-oracle merged commit a0da657 into blind-oracle:main Sep 21, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants