Postgres changes into Kinesis Data Streams.
One record per change, partitioned by the row's ordering key so each row's history stays on one shard, in order.
The Kinesis sink writes the change JSON as record data and uses the ordering key as partition key. Consumers — Lambda, Kinesis Data Analytics, Firehose — read shards in order, so per-row sequencing comes free.
How delivery works
| Aspect | Behaviour for Amazon Kinesis |
|---|---|
| Record | Data = the change JSON; PartitionKey = ordering key (truncated to 256 characters). |
| Batching | One PutRecords call per batch; keep the batch at or under 500 records. |
| Ordering | Same partition key → same shard, in commit order. |
| Idempotency | The idempotency_key inside the record. |
| On failure | Any failed record retries the whole batch. A failed request is retried up to 5 times with exponential backoff (30 s, 1, 2, 4 min, with jitter). After the fifth failure the message is parked as undelivered, the console shows it with the last error, and one click replays it — later changes to the same row wait behind it so order is preserved. |
| Connection check | DescribeStreamSummary on the stream. |
Set up in three steps
-
Connect your database
Paste a connection string and press Test. Waltail checks the version (12+), that
wal_levelislogical, that the user may replicate, and that a slot is free — and shows the fix for anything that fails. Then it creates its own publication and replication slot. Connection guide → -
Add Amazon Kinesis as a destination
Pick the tables to stream, choose Amazon Kinesis, and fill in Stream name, Region, Access key ID, Secret access key. Press Test — Waltail checks it can reach and write to Amazon Kinesis before anything is saved.
-
Change a row
Insert or update a row. The console shows the first event as it is captured and delivered; from then on, every committed change follows within about a second. Have rows that already exist? Run a backfill — confirm the estimate and they arrive through Amazon Kinesis as
readevents.
Configuration
Fields as the console asks for them. Secrets are encrypted at rest and never shown again.
| Field | Console label | Required | Notes |
|---|---|---|---|
stream_name | Stream name | Yes | |
region | Region | Yes | AWS region of the resource, e.g. us-east-1. |
access_key_id | Access key ID | Yes | Static IAM credentials; Waltail does not assume roles. |
secret_access_key | Secret access key | Yes | Stored encrypted, never shown again. |
endpoint | Custom endpoint | Optional | Overrides the AWS endpoint for VPC endpoints or LocalStack. |
Every sink also has two tuning settings: batch size (messages per request, default 100) and rate limit (messages per second, default unlimited). Changes to one row are never in flight twice at once.
Example
Every change is this JSON envelope — data is the row after the change, old the row before it when replica identity provides it, null otherwise:
{
"op": "update",
"table": "public.orders",
"data": { "id": 42, "status": "shipped", "total": 19.99 },
"old": { "id": 42, "status": "paid", "total": 19.99 },
"seq": 137,
"idempotency_key": "9f1c3d2e-6b4a-4c7e-9d0f-2a1b3c4d5e6f",
"commit_lsn": "0/1A2B3C8"
}Questions
Why might I see a record twice?
A batch that partly failed is retried whole, so the records that did land appear again. Dedupe on idempotency_key.
Which permissions are needed?
kinesis:PutRecords and kinesis:DescribeStreamSummary on the stream.