[Backport 7.71.x] [AI-6101] Add support for customizable cache keys for persistent cache

[datadog-agent-integrations-bot]

Backport fc7560f from #21316.

---

What does this PR do?

Original implementation

The implementation outlined here is the original idea when the PR was opened. After review, the implementation has changed quite a bit.

This PR adds support for customizable cache keys to be used with the agent persistent cache. This is implemented in the following way:

• A CacheKey abstract class that can be extended to implement any cache invalidation logic needed.
• The AgentCheck has been modified to rely on a CacheKey implementation to provide the key for the persistent cache internally. Using check_id is now the default behavior through the FullConfigCacheKey implementation but can be overriden when necessary.
• The logs_persistent_cache_key can be overriden by a developer that wants to modify what is the CacheKey used to handle the logs cursor.
• We are still maintaining some level of separation between checks by adding the check name to the key. It is not the full check_id but, if the developer creates a CacheKey that is to simple, the impact of key clashing is reduced to the same kind of check.
• Added 2 implementations of the CacheKey interface that work as an example and to provide funcionality:
  • The FullConfigCacheKey is equivalent to the previous behavior. It relies on the check_id that includes a digest of the full configuration of the check. This default behavior makes this change backward compatible with the usage of send_log and get_log_cursor, both of them using persistent cache.
  • The ConfigSetCacheKey implementation invalidates the cache only when a subset of configuration options have been updated. This provides a similar level of separation between different checks but allowing some configuration options to not invalidate the cache.
• The CacheKeyManager allows multiple cache keys to coexist in the AgentCheck for differnet purposes. At the moment we only use the CacheKey internally to manage the log cursor. However, if we want to include something else we can register a new type of CacheKeyType enum and maintain the new one in the manager without having to modify how the AgentCheck behaves.

Note: the manager forces to have each cache key represented by an enum to avoid relying on simple string keys per cache key. While the intention is to allow to personalize the cache invalidation mechanism, we do not want potentially arbitrary cache keys being register which could trigger cache misses if the string key is mistakenly generated.

This PR adds support for customizing the persistent cache prefix used when caching any data in the AgentCheck. This is done by allowing the developer to override the persistent_cache_id method. This method returns the ID that identifies the specific check instance in the Agent persistent cache.

When a given check returns the same persistent cache ID, the cache keys generated will be stable. By default, the persistent cache ID is the digest of the full check configuration. Any update in the configuration will generate a different ID, causing the cache to be invalidated. This is the current behavior before introducing this feature, which ensures backward compatibility.

Motivation

Until now, integrations that wanted to emit logs and be consistent between agent restarts would have logs missing or duplicated if the user changed their configuration. This would invalidate the cache and there was limited control over the last cursor.

The intention of this change is to provide a backward compatible solution that allows developers to modify the cache invalidation behavior in an easy way.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[codecov]

Codecov Report

❌ Patch coverage is 98.44961% with 2 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (7.71.x@865b063). Learn more about missing BASE report.

Additional details and impacted files

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.