Skip to content

Conversation

yasicar
Copy link

@yasicar yasicar commented Sep 1, 2025

PR Summary: Enhanced AWS Resource Metadata Support

Enhancement Overview

This PR adds enhanced AWS resource metadata support to YACE (Yet Another CloudWatch Exporter), extending its capabilities beyond basic tag collection to include service-specific attributes.

DynamoDB Enhancement:

  • YACE already had excellent support for discovering DynamoDB tables and collecting their CloudWatch metrics with tags
  • This enhancement adds extra support for collecting valuable table attributes such as:
    • table_class (Standard vs Standard-IA)
    • billing_mode (Pay-per-request vs Provisioned)
    • table_status (Active, Creating, Deleting, etc.)
  • This enables more sophisticated Prometheus queries that can filter and group by these operational attributes

Solution Architecture

Implemented a flexible and extensible AWS resource metadata collection framework that:

  1. Extended TaggedResource struct with a new Metadata field (map[string]string) to store service-specific attributes alongside existing tags

  2. Built a scalable service filter architecture using the ServiceFilters map, allowing each AWS service namespace to define custom metadata collection logic

  3. Added DynamoDB metadata collection as the first implementation example:

    "AWS/DynamoDB": {
        FilterFunc: func(ctx context.Context, client client, inputResources []*model.TaggedResource) ([]*model.TaggedResource, error) {
            // Uses DynamoDB DescribeTable API to fetch additional table metadata
            // Stores table_class and other attributes in resource.Metadata
        }
    }
  4. Enhanced Prometheus metric generation to export metadata as additional labels in info metrics alongside existing tag labels

Benefits

  • Enhanced DynamoDB monitoring - Users can now create advanced Prometheus queries like:
    aws_dynamodb_consumed_read_capacity_units{table_class="Standard"}
  • Generic extensible framework - Future contributors can easily add metadata collection for any AWS service by implementing a FilterFunc
  • Fully backward compatible - All existing configurations continue to work exactly as before
  • Efficient implementation - Metadata collection occurs during resource discovery phase, maintaining performance

Enhanced Output

Before:

aws_dynamodb_info{name="arn:aws:dynamodb:us-east-1:123:table/MyTable", tag_Environment="prod"}

After (with additional metadata):

aws_dynamodb_info{name="arn:aws:dynamodb:us-east-1:123:table/MyTable", tag_Environment="prod", table_class="Standard"}

This enhancement provides a solid foundation for extending metadata collection to other AWS services such as EC2 instance types, RDS engine versions, S3 bucket classes, and more, making YACE even more powerful for comprehensive AWS monitoring.

@yasicar yasicar changed the title feature-aws-resource-metadata [FEATURE] feature-aws-resource-metadata Sep 8, 2025
@yasicar yasicar changed the title [FEATURE] feature-aws-resource-metadata [FEATURE] Enhanced AWS Resource Metadata Support Sep 8, 2025
@yasicar
Copy link
Author

yasicar commented Sep 13, 2025

@cristiangreco Can you please help me with the review
PR: #1732

@yasicar yasicar force-pushed the feature-aws-resource-metadata branch from cbf4898 to 79d91a8 Compare September 29, 2025 14:33
@SuperQ SuperQ requested a review from cristiangreco October 7, 2025 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant