- Home
- Destinations
- Apache Kafka
Postgres to Kafka, with nothing to run.
Hosted change data capture into any broker that speaks the Kafka protocol — Amazon MSK, Redpanda, Confluent Cloud, Aiven, WarpStream, Upstash — with the row's primary key as the record key.
The Kafka sink replaces a self-managed connector cluster with a connection string. Waltail reads the write-ahead log, and produces one record per change to your topic with the ordering key as the record key, so every change to a row lands on the same partition in commit order.
Waltail does not run a Kafka cluster for you — you bring the broker. That is the point: your data goes straight to the system your consumers already read from.
How delivery works
| Aspect | Behaviour for Apache Kafka |
|---|---|
| Record | Value = the change JSON; key = the ordering key (schema.table:pk); header idempotency_key. |
| Batching | One produce per batch (default 100 records), acknowledged by the broker before the batch is marked delivered. |
| Ordering | Same key → same partition, in commit order. Use the key for log compaction if you keep a current-state topic. |
| Idempotency | Dedupe on the idempotency_key header or the field inside the value. |
| On failure | A failed request is retried up to 5 times with exponential backoff (30 s, 1, 2, 4 min, with jitter). After the fifth failure the message is parked as undelivered, the console shows it with the last error, and one click replays it — later changes to the same row wait behind it so order is preserved. |
| Connection check | A metadata round-trip to the brokers with the configured TLS and SASL settings. |
Set up in three steps
-
Connect your database
Paste a connection string and press Test. Waltail checks the version (12+), that
wal_levelislogical, that the user may replicate, and that a slot is free — and shows the fix for anything that fails. Then it creates its own publication and replication slot. Connection guide → -
Add Apache Kafka as a destination
Pick the tables to stream, choose Apache Kafka, and fill in Bootstrap brokers, Topic. Press Test — Waltail checks it can reach and write to Apache Kafka before anything is saved.
-
Change a row
Insert or update a row. The console shows the first event as it is captured and delivered; from then on, every committed change follows within about a second. Have rows that already exist? Run a backfill — confirm the estimate and they arrive through Apache Kafka as
readevents.
Configuration
Fields as the console asks for them. Secrets are encrypted at rest and never shown again.
| Field | Console label | Required | Notes |
|---|---|---|---|
brokers | Bootstrap brokers | Yes | Comma-separated host:port list. |
topic | Topic | Yes | Auto-created when the broker allows it. |
tls | TLS | Optional | Hosted brokers require it. |
ca_cert | CA certificate | Optional | PEM for private CAs; implies TLS. |
sasl_mechanism | SASL mechanism | Optional | plain, scram-sha-256 or scram-sha-512. |
sasl_username | SASL username | Optional | |
sasl_password | SASL password | Optional |
Every sink also has two tuning settings: batch size (messages per request, default 100) and rate limit (messages per second, default unlimited). Changes to one row are never in flight twice at once.
Example
Every change is this JSON envelope — data is the row after the change, old the row before it when replica identity provides it, null otherwise:
{
"op": "update",
"table": "public.orders",
"data": { "id": 42, "status": "shipped", "total": 19.99 },
"old": { "id": 42, "status": "paid", "total": 19.99 },
"seq": 137,
"idempotency_key": "9f1c3d2e-6b4a-4c7e-9d0f-2a1b3c4d5e6f",
"commit_lsn": "0/1A2B3C8"
}Questions
Do I need a connector framework?
No. Waltail connects to your Postgres over logical replication and produces directly to the broker. There is no connector cluster or JVM in the path.
Which brokers work?
Anything that speaks the Kafka protocol: Apache Kafka, Amazon MSK, Redpanda, Confluent Cloud, Aiven, WarpStream, Upstash. Azure Event Hubs has its own sink that fills in the protocol details for you.
What is the message format?
A plain envelope — op, table, data, old, sequence, idempotency key and commit LSN. Consumers written against another tool's envelope need a small mapping; see the message format reference.
Can one topic carry many tables?
Yes. The table field names the source table, and the key is prefixed with it, so consumers can route per table.