- Home
- Destinations
- Google Pub/Sub
Postgres changes into Google Pub/Sub.
Pub/Sub's ordering key is Waltail's ordering key, so per-row order is enforced by Pub/Sub itself — for Cloud Run, Cloud Functions, Dataflow and everything else that subscribes.
The Pub/Sub sink publishes one message per change with ordering enabled: the row's ordering key maps straight onto Pub/Sub's ordering key, the change JSON is the data, and the idempotency key is an attribute. Works with any Postgres, including Cloud SQL and AlloyDB.
How delivery works
| Aspect | Behaviour for Google Pub/Sub |
|---|---|
| Message | Data = the change JSON; OrderingKey = ordering key; attribute idempotency_key. |
| Batching | All messages of a batch are published, then every publish result is awaited before the batch is marked delivered. |
| Ordering | Native: subscriptions with ordering enabled deliver each row's changes in commit order. |
| Idempotency | Dedupe on the idempotency_key attribute. |
| On failure | A failed request is retried up to 5 times with exponential backoff (30 s, 1, 2, 4 min, with jitter). After the fifth failure the message is parked as undelivered, the console shows it with the last error, and one click replays it — later changes to the same row wait behind it so order is preserved. |
| Connection check | Confirms the topic exists and the key can see it. |
Set up in three steps
-
Connect your database
Paste a connection string and press Test. Waltail checks the version (12+), that
wal_levelislogical, that the user may replicate, and that a slot is free — and shows the fix for anything that fails. Then it creates its own publication and replication slot. Connection guide → -
Add Google Pub/Sub as a destination
Pick the tables to stream, choose Google Pub/Sub, and fill in Project ID, Topic, Service account key (JSON). Press Test — Waltail checks it can reach and write to Google Pub/Sub before anything is saved.
-
Change a row
Insert or update a row. The console shows the first event as it is captured and delivered; from then on, every committed change follows within about a second. Have rows that already exist? Run a backfill — confirm the estimate and they arrive through Google Pub/Sub as
readevents.
Configuration
Fields as the console asks for them. Secrets are encrypted at rest and never shown again.
| Field | Console label | Required | Notes |
|---|---|---|---|
project_id | Project ID | Yes | |
topic | Topic | Yes | Create it first; Waltail does not create topics. |
credentials_json | Service account key (JSON) | Yes | Needs pubsub.publisher on the topic. |
Every sink also has two tuning settings: batch size (messages per request, default 100) and rate limit (messages per second, default unlimited). Changes to one row are never in flight twice at once.
Example
Every change is this JSON envelope — data is the row after the change, old the row before it when replica identity provides it, null otherwise:
{
"op": "update",
"table": "public.orders",
"data": { "id": 42, "status": "shipped", "total": 19.99 },
"old": { "id": 42, "status": "paid", "total": 19.99 },
"seq": 137,
"idempotency_key": "9f1c3d2e-6b4a-4c7e-9d0f-2a1b3c4d5e6f",
"commit_lsn": "0/1A2B3C8"
}Questions
Do I need to enable ordering on the subscription?
Yes — ordering is a subscription setting in Pub/Sub. Waltail always publishes with an ordering key.
Which roles does the service account need?
roles/pubsub.publisher on the topic, plus pubsub.topics.get for the connection check (included in Viewer or Publisher at the topic level).