Fan out Postgres changes with Amazon SNS.
Publish each change once and let SNS deliver it to every subscriber — queues, Lambda, HTTP, email. FIFO topics keep per-row order.
The SNS sink publishes each change to a topic. Use it when several services need the same stream: one pipeline, one sink, and SNS subscriptions decide who receives what, optionally filtered on the attributes Waltail sets.
How delivery works
| Aspect | Behaviour for Amazon SNS |
|---|---|
| Message | Message = the change JSON; attributes idempotency_key and ordering_key. |
| Batching | PublishBatch in chunks of 10. |
| Ordering | FIFO topics: MessageGroupId = ordering key. Standard topics: best effort. |
| Idempotency | FIFO topics: MessageDeduplicationId = idempotency key. |
| On failure | A partial failure fails and retries the batch. A failed request is retried up to 5 times with exponential backoff (30 s, 1, 2, 4 min, with jitter). After the fifth failure the message is parked as undelivered, the console shows it with the last error, and one click replays it — later changes to the same row wait behind it so order is preserved. |
| Connection check | GetTopicAttributes on the ARN. |
Set up in three steps
-
Connect your database
Paste a connection string and press Test. Waltail checks the version (12+), that
wal_levelislogical, that the user may replicate, and that a slot is free — and shows the fix for anything that fails. Then it creates its own publication and replication slot. Connection guide → -
Add Amazon SNS as a destination
Pick the tables to stream, choose Amazon SNS, and fill in Topic ARN, Region, Access key ID, Secret access key. Press Test — Waltail checks it can reach and write to Amazon SNS before anything is saved.
-
Change a row
Insert or update a row. The console shows the first event as it is captured and delivered; from then on, every committed change follows within about a second. Have rows that already exist? Run a backfill — confirm the estimate and they arrive through Amazon SNS as
readevents.
Configuration
Fields as the console asks for them. Secrets are encrypted at rest and never shown again.
| Field | Console label | Required | Notes |
|---|---|---|---|
topic_arn | Topic ARN | Yes | A .fifo ARN switches on group and deduplication ids. |
region | Region | Yes | AWS region of the resource, e.g. us-east-1. |
access_key_id | Access key ID | Yes | Static IAM credentials; Waltail does not assume roles. |
secret_access_key | Secret access key | Yes | Stored encrypted, never shown again. |
endpoint | Custom endpoint | Optional | Overrides the AWS endpoint for VPC endpoints or LocalStack. |
Every sink also has two tuning settings: batch size (messages per request, default 100) and rate limit (messages per second, default unlimited). Changes to one row are never in flight twice at once.
Example
Every change is this JSON envelope — data is the row after the change, old the row before it when replica identity provides it, null otherwise:
{
"op": "update",
"table": "public.orders",
"data": { "id": 42, "status": "shipped", "total": 19.99 },
"old": { "id": 42, "status": "paid", "total": 19.99 },
"seq": 137,
"idempotency_key": "9f1c3d2e-6b4a-4c7e-9d0f-2a1b3c4d5e6f",
"commit_lsn": "0/1A2B3C8"
}Questions
Can subscribers filter by table?
The table name is inside the message body, not an attribute, so filter policies can match on ordering_key (which starts with schema.table:) using a prefix filter, or subscribe everything and branch in the consumer.
Which permissions are needed?
sns:Publish and sns:GetTopicAttributes on the topic.