Add integration tests

[spompea-cfa]

Add integration tests plus README instructions on how to run them.

Additional ancillary updates:

• Update README to enumerate all environment variables used by middleware.
• Exporter puts the output file in a specific (configurable) folder and uses a chronologically sortable naming format: 2025-10-07T23:15:54-UTC-exporter-output.json (the tag, e.g., exporter-output, is also configurable).
• Other tidying.

[Copilot]

Pull Request Overview

This PR adds comprehensive testing infrastructure and documentation, along with several improvements to the middleware system. The changes include a full integration test that validates the entire data flow from SQS ingestion through Senzing processing to S3 export, plus configuration enhancements and documentation updates.

Key changes:

• Integration test framework using unittest with LocalStack services
• Enhanced exporter with configurable output folder and chronologically sortable filename format
• Comprehensive README updates with environment variable documentation and testing instructions

Reviewed Changes

Copilot reviewed 8 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/test_flow.py	Full integration test validating SQS → Consumer → Redoer → Exporter → S3 flow
test/fixtures/flow-output.json	Expected JSON output for test validation
test/fixtures/customers.jsonl	Test data containing 120 customer records
middleware/exporter.py	Enhanced with configurable folder and timestamp-based filenames
middleware/redoer.py	Added documentation comments explaining retry logic
docker-compose.yaml	Improved service dependencies and environment variable organization
dev-scripts/s3_get.py	Fixed file path handling for S3 folder structure
README.md	Comprehensive documentation of environment variables and testing procedures

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[jamesiarmes]

Can you add a GitHub actions workflow to run this automatically on push?

[spompea-cfa]

Can you add a GitHub actions workflow to run this automatically on push?

good idea. will open a separate PR.