Runs a background task that constantly monitors for messages in an AWS SQS queue, the messages are then stored in
the dynamic database in a table with the specified name. The table columns are based on the properties of the message in the SQS queue.
Messages are batched before being written to the database for better performance.
AWS SQS Setup Requirements:
1. AWS Credentials Configuration:
The container automatically discovers AWS credentials using the AWS SDK default credential chain.
For ECS deployments:
- Create an IAM Role with the SQS permissions listed below
- Assign this role as the 'Task Role' (not Task Execution Role) in your ECS Task Definition
- The container will automatically assume this role - no environment variables needed
For Non AWS deployments:
- Set environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
2. Required IAM Permissions:
The IAM role/user must have these SQS permissions:
- sqs:ReceiveMessage - to receive messages from the queue
- sqs:DeleteMessage - to delete processed messages from the queue
- sqs:GetQueueAttributes - to retrieve queue metadata
Example IAM policy:
{
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:GetQueueAttributes"
],
"Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:QUEUE_NAME"
}
3. Queue Configuration:
- Messages in the queue must be in JSON format
- Consider setting appropriate visibility timeout on the queue to prevent message reprocessing
4. Table Schema:
- On first run, the table will be created automatically with columns based on the first batch of messages
- Column sizes are calculated dynamically with a 20% buffer (min 100 chars, max 8000 chars)
- If new fields appear in later messages, columns will be added automatically
- The table will always use 'sqs_message_id' as the primary key
- Set include_metadata=true to also store _sqs_receipt_handle and _sqs_received_at fields
Tables Affected
Permissions needed
- AWS permission and policies