SQS Entity Resolution using Senzing

This is an implementation of entity resolution using Senzing with AWS SQS. Data is sent to an SQS queue, which is processed by a "consumer" service that forwards the data to Senzing for entity resolution. The results can then be exported to an S3 bucket.

architecture-beta
  group vpc(cloud)[VPC]
  group ecs(cloud)[ECS Fargate] in vpc

  service queue(database)[SQS Queue] in vpc
  service consumer(server)[Senzing Consumer] in ecs
  service exporter(server)[Senzing Exporter] in ecs
  service db(database)[PostgreSQL] in vpc
  service s3(disk)[S3 Bucket]

  consumer:L --> T:queue
  consumer:B --> T:db
  exporter:B --> R:db
  exporter:R --> L:s3

Loading

Local development with Docker

This repository includes a docker-compose.yml file that can be used to develop and run the consumer service on our local machine. This setup includes:

• SQS and S3 emulators using LocalStack
  • An S3 bucket named sqs-senzing-local-export
  • An SQS queue named sqs-senzing-local-ingest
  • An SQS queue named sqs-senzing-local-redo
• A local PostgreSQL database
• A database initialization container to set up the Senzing schema
• The Senzing consumer service (in development)
• A tools container with the Senzing v4 SDK and awslocal wrapper for interacting with LocalStack services

Starting the services

1. Build the necessary images:

   docker compose build

2. Start the services:

   docker compose up -d

3. Access the tools container to interact with the services:

   docker compose run tools /bin/bash

Using the services

The tools container should be configured with the necessary environment variables to interact with the SQS and S3 services in LocalStack, as well as the Senzing SDK.

You can use the awslocal command to interact with the SQS and S3 services. For example, to send a message to the SQS queue:

awslocal sqs send-message \
  --queue-url http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/sqs-senzing-local-ingest \
  --message-body '{"NAME_FULL":"Robert Smith", "DATE_OF_BIRTH":"7/4/1976", "PHONE_NUMBER":"555-555-2088"}'

You can use the Senzing SDK's sz_* commands to interact with the Senzing database. For example, to add a new entity:

sz_command -C add_record \
  PEOPLE 1 '{"NAME_FULL":"Robert Smith", "DATE_OF_BIRTH":"7/4/1976", "PHONE_NUMBER":"555-555-2088"}'