S3 Data Deletion

[MJawad-AbouAleiwi]

Hello,

I'm using Pyroscope tool, and I'm using it with S3 Object Storage. How can I delete the data in this bucket after 30 days for example? Which parameter is responsible for this?

Thank you.

[korniltsev]

The parameter responsible for deleting data in S3 after a specified retention period is:

-compactor.blocks-retention-period

You can set this to 720h (30 days) to delete data older than 30 days.

Configuration options:

1. Command line flag:
   -compactor.blocks-retention-period=720h
2. Configuration file (YAML):

  compactor:
    compactor_blocks_retention_period: 720h

4. Per-tenant limits (if using multi-tenancy):

  limits:
    # Global default
    compactor_blocks_retention_period: 720h

  # Per-tenant overrides
  limits_overrides:
    tenant-id:
      compactor_blocks_retention_period: 720h

Notes:

• Default value is 0s (disabled - no automatic deletion)
• The compactor component handles the retention cleanup
• Setting it to 0 disables automatic retention-based deletion
• The retention is applied based on the block's MaxTime field

[korniltsev]

The compactor.deletion_delay parameter controls how long to wait before actually deleting blocks that are marked for deletion. This is a safety mechanism that provides a grace period.

Here's how both parameters work together:

1. compactor.blocks-retention-period=720h (30 days) - This marks blocks for deletion after they exceed the retention period
2. compactor.deletion_delay - This adds an additional delay before the marked blocks are actually deleted from S3

The default deletion_delay is typically 48 hours, which means:

• After 30 days (720h), blocks are marked for deletion
• After an additional 48 hours, they are permanently deleted from S3

This two-stage deletion process helps prevent accidental data loss. If you need immediate deletion without the grace period, you can set compactor.deletion_delay=0, but this is not
recommended for production environments.

Configuration example:
compactor:
blocks_retention_period: 720h # 30 days
deletion_delay: 48h # Additional 2-day grace period

Or via command line flags:
-compactor.blocks-retention-period=720h -compactor.deletion-delay=48h

[MJawad-AbouAleiwi]

Thank you very much for your answer and for explanation! That helps a lot!
I still have a small question regarding Grafana Alloy, if I'm using the Pyroscope SDKs to profile applications and then forward the profiles toward a Central Alloy that sends the profiles to the server, is there any component in alloy to down sample the received profiles per service? Do you think of this idea to be added in future releases?

[korniltsev]

you can try using pyroscope.relabel to drop some profiles, for example using hashmod relabeling rule

[MJawad-AbouAleiwi]

For example, is this the correct way to drop 30% of profiles per service (30% of profiles of each service, not 30% of the services):

pyroscope.relabel "filter_profiles" {
forward_to = [pyroscope.write.backend.receiver]
rule {
action = "hashmod"
source_labels = ["service_name", "name"]
target_label = "__tmp_hash"
modulus = 10
}

rule {
    action = "drop"
    source_labels = ["__tmp_hash"]
    regex = "^(0|1|2)$"
}

}

[korniltsev]

yes, looks good from the first glance. did not test/check though.