- Home
- Destinations
- Amazon S3
Every Postgres change, archived to S3.
An append-only record of what changed, as NDJSON objects under dated prefixes — for audit trails, compliance retention, and feeding a lakehouse later.
The S3 sink writes each delivered batch as one object of newline-delimited change JSON under prefix/YYYY/MM/DD/. The batch size is the file-size dial: larger batches mean fewer, bigger objects. Path-style addressing makes MinIO and other S3-compatible stores work too.
How delivery works
| Aspect | Behaviour for Amazon S3 |
|---|---|
| Object | Key [prefix/]YYYY/MM/DD/<pipeline>-<firstSeq>-<lastSeq>.ndjson, Content-Type: application/x-ndjson, one change per line in commit order. |
| Batching | One object per batch (default 100 changes). Raise the batch size for fewer, larger files. |
| Ordering | Lines within an object and the sequence numbers in the key are in commit order. |
| Idempotency | A retried batch rewrites the same key, so retries never duplicate objects. |
| On failure | A failed request is retried up to 5 times with exponential backoff (30 s, 1, 2, 4 min, with jitter). After the fifth failure the message is parked as undelivered, the console shows it with the last error, and one click replays it — later changes to the same row wait behind it so order is preserved. |
| Connection check | Writes a small marker object [prefix/].waltail-check to prove write permission. |
Set up in three steps
-
Connect your database
Paste a connection string and press Test. Waltail checks the version (12+), that
wal_levelislogical, that the user may replicate, and that a slot is free — and shows the fix for anything that fails. Then it creates its own publication and replication slot. Connection guide → -
Add Amazon S3 as a destination
Pick the tables to stream, choose Amazon S3, and fill in Bucket, Region, Access key ID, Secret access key. Press Test — Waltail checks it can reach and write to Amazon S3 before anything is saved.
-
Change a row
Insert or update a row. The console shows the first event as it is captured and delivered; from then on, every committed change follows within about a second. Have rows that already exist? Run a backfill — confirm the estimate and they arrive through Amazon S3 as
readevents.
Configuration
Fields as the console asks for them. Secrets are encrypted at rest and never shown again.
| Field | Console label | Required | Notes |
|---|---|---|---|
bucket | Bucket | Yes | |
prefix | Key prefix | Optional | e.g. waltail/prod; a dated path is appended. |
use_path_style | Path-style addressing | Optional | For MinIO and compatible stores. |
region | Region | Yes | AWS region of the resource, e.g. us-east-1. |
access_key_id | Access key ID | Yes | Static IAM credentials; Waltail does not assume roles. |
secret_access_key | Secret access key | Yes | Stored encrypted, never shown again. |
endpoint | Custom endpoint | Optional | Overrides the AWS endpoint for VPC endpoints or LocalStack. |
Every sink also has two tuning settings: batch size (messages per request, default 100) and rate limit (messages per second, default unlimited). Changes to one row are never in flight twice at once.
Example
Every change is this JSON envelope — data is the row after the change, old the row before it when replica identity provides it, null otherwise:
{
"op": "update",
"table": "public.orders",
"data": { "id": 42, "status": "shipped", "total": 19.99 },
"old": { "id": 42, "status": "paid", "total": 19.99 },
"seq": 137,
"idempotency_key": "9f1c3d2e-6b4a-4c7e-9d0f-2a1b3c4d5e6f",
"commit_lsn": "0/1A2B3C8"
}Questions
Can I query the archive?
Yes — NDJSON under dated prefixes is the layout Athena, DuckDB, Spark and most lakehouse loaders read directly.
Does it work with MinIO or Cloudflare R2?
Any S3-compatible store with a custom endpoint works; turn on path-style addressing for MinIO.
Which permissions are needed?
s3:PutObject on the prefix.