feat: feeders as config

[KayakinCoder]

Related issue(s)

SNS and S3 are common event sources for Cirrus Payloads. This feature provides a config-driven means to feed those event sources to the Cirrus Process Queue.

Proposed Changes

1. A cirrus_inputs.feeder_definitions_dir option to denote a directory containing N number of subdirectories, each of which defines a feeder (similar to cirrus_inputs.task_definitions_dir). For example:

/cirrus
    /feeders
        /feeder1
            definition.yaml
        /feeder2
            definition.yaml

Each definition.yaml creates:

• N number of SNS and/or S3 triggers
  • Note that this module does not create SNS topics or S3 buckets. Rather, it provides a way to generate the necessary AWS permissions and resources for existing topics and/or buckets to trigger messages to an SQS queue
• A single SQS queue to receive messages from those trigger(s)
• A single lambda function to read messages from that SQS queue, process them, and send them to the Cirrus Process Queue

See /modules/cirrus/cirrus-examples/feeders and /modules/cirrus/feeder/README.md for more detail

2. Additionally, a submodule /modules/cirrus/feeder/lambda was created as a potential for DRY-ing up task + feeder lambda creation. Both tasks and feeders require (quite involved) lambda creation; a single lambda module that both can use is the future hope.

Note: Feeders should be considered as being in beta, not production-ready quite yet. A few remaining tasks items will be buttoned up in a future PR:

• Full SSM support. See feeder_task_workflow.tf todo
• Detailed documentation in /modules/cirrus/cirrus-examples/feeders

Testing

This change was validated by the following observations:

1. Build FilmDrop with feeder_definitions_dir = null to ensure no changes to existing behavior
2. Build FilmDrop with feeder_definitions_dir = /somedir and various feeder configuration options to validate expected feeder behavior

Checklist

• I have deployed and validated this change
• Changelog
  • I have added my changes to the changelog
  • No changelog entry is necessary
• README migration
  • I have added any migration steps to the Readme
  • No migration is necessary

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[KayakinCoder]

An easy-ish way to test creation of feeder infrastructure:

• Clone https://github.com/Element84/temp-filmdrop-testing
• tfenv install
• Edit theproject_name var in filmdrop.tfvars (AWS infra that's created will contain this string, for searchability)
• Auth to an AWS account that has the requisite Control Tower-created VPC which FilmDrop expects
• terraform init
• terraform apply -var-file=filmdrop.tfvars