- Home
- Destinations
- Elasticsearch
Keep Elasticsearch in sync with Postgres.
Row changes become bulk index and delete operations with the primary key as document id. Under a second from commit to searchable, with no indexing pipeline or cron re-index.
The Elasticsearch sink is the clearest demonstration of low-latency CDC: a row is inserted or updated and the document is upserted; a row is deleted and the document is removed. Each batch is one _bulk request, and because the document id is derived from the primary key, retries are idempotent and updates overwrite in place.
How delivery works
| Aspect | Behaviour for Elasticsearch |
|---|---|
| Operation | Insert/update → index with _id = sanitised ordering key and the row's data as the document; delete → delete of the same _id. |
| Batching | One _bulk NDJSON request per batch. |
| Ordering | Per-row order is preserved, so an update never lands after the delete that followed it. |
| Idempotency | Indexing by _id is idempotent; a retried batch overwrites the same documents. |
| On failure | Any item error other than a 404 on delete fails the batch. Any 2xx is accepted. A 429 pauses the sink for the Retry-After period (30 s if absent) without spending a retry or metering egress. A failed request is retried up to 5 times with exponential backoff (30 s, 1, 2, 4 min, with jitter). After the fifth failure the message is parked as undelivered, the console shows it with the last error, and one click replays it — later changes to the same row wait behind it so order is preserved. |
| Connection check | GET / on the cluster with the configured credentials. |
Set up in three steps
-
Connect your database
Paste a connection string and press Test. Waltail checks the version (12+), that
wal_levelislogical, that the user may replicate, and that a slot is free — and shows the fix for anything that fails. Then it creates its own publication and replication slot. Connection guide → -
Add Elasticsearch as a destination
Pick the tables to stream, choose Elasticsearch, and fill in Cluster URL, Index. Press Test — Waltail checks it can reach and write to Elasticsearch before anything is saved.
-
Change a row
Insert or update a row. The console shows the first event as it is captured and delivered; from then on, every committed change follows within about a second. Have rows that already exist? Run a backfill — confirm the estimate and they arrive through Elasticsearch as
readevents.
Configuration
Fields as the console asks for them. Secrets are encrypted at rest and never shown again.
| Field | Console label | Required | Notes |
|---|---|---|---|
endpoint | Cluster URL | Yes | |
index | Index | Yes | Created by the first write if it does not exist. |
username | Username | Optional | Basic auth. |
password | Password | Optional | |
api_key | API key | Optional | Sent as Authorization: ApiKey; takes precedence over basic auth. |
Every sink also has two tuning settings: batch size (messages per request, default 100) and rate limit (messages per second, default unlimited). Changes to one row are never in flight twice at once.
Example
What Elasticsearch receives:
{"index":{"_id":"public_orders_42"}}
{"id":42,"status":"shipped","total":19.99}
{"delete":{"_id":"public_orders_43"}}Questions
What is the document id?
The ordering key (schema.table:pk) with characters outside [A-Za-z0-9_-] replaced by _, e.g. public_orders_42. Composite keys join their values.
Can I control the mapping?
Create the index with your mapping before adding the sink; Waltail only writes documents. Dynamic mapping handles the rest if you don't.
Does it work with Elastic Cloud?
Yes — use the cluster endpoint and an API key.