Sign in
  1. Home
  2. Docs
  3. Backfills
DOCS

Backfills.

Streaming delivers what changes from now on. A backfill delivers the rows that already exist — through the same pipeline, to every destination, in an order that stays correct next to live changes.

When you want one

  • A new destination needs the past — you are seeding a search index, a cache, or a consumer that builds state from every row.
  • You added a table to a running pipeline. Changes from before you added it were never captured, so no replay can produce them. The console offers a backfill the moment you add tables.
  • You rebuilt a consumer and want a full current picture of a table rather than a window of recent history.

How to run one

  1. Pick the tables

    Open the pipeline and go to the Backfills tab. Tick any of the pipeline's tables. A table needs a primary key or a replica-identity index so Waltail can read it in a stable order; the console marks any table it cannot backfill and says why.

  2. Read the estimate, then confirm

    Waltail looks at your database and shows roughly how many rows and gigabytes the backfill will deliver. Backfilled rows count as egress like any other delivery, so nothing starts until you press Confirm on that number.

  3. Watch it run

    A worker reads the tables in pages of 1,000 rows. It stores each page and moves a per-table cursor forward in the same transaction, so if a worker restarts it picks up exactly where it stopped — no gaps, no duplicates. You can follow the progress per table in the console, and cancel at any time; what was already delivered stays delivered.

What your destinations receive

Each row arrives as an ordinary message with op: "read", the whole row in data, no old, and commit_lsn of 0/0 — see the message format. Treat a read as an upsert: search sinks index it, the Redis cache sets it, queues and webhooks carry it like any insert.

How it stays in order with live changes

Backfilled rows use the same ordering key as live changes and take their sequence numbers from the same counter. So a row's snapshot always sorts after every change captured before the backfill read it, and before every change captured after. Your consumer never sees an old snapshot overwrite a newer update.

Good to know

  • One backfill runs per pipeline at a time; start another when it finishes.
  • Only the pipeline's own tables can be backfilled. A snapshot of a table the pipeline does not capture would never be followed by its changes.
  • A backfill runs while the pipeline is running. Pause the pipeline and the backfill pauses too.
  • Included on every plan, Free too. Metered as egress at your plan's rates.

From the API

# 1. ask for the estimate
curl -s https://app.waltail.com/api/pipelines/PIPELINE_ID/backfills/estimate \
  -H "Authorization: Bearer pgcdc_…" -H "Content-Type: application/json" \
  -d '{"tables": ["public.orders"]}'

# 2. start it, confirming the metered volume
curl -s https://app.waltail.com/api/pipelines/PIPELINE_ID/backfills \
  -H "Authorization: Bearer pgcdc_…" -H "Content-Type: application/json" \
  -d '{"tables": ["public.orders"], "confirm": true}'

Start streaming in minutes.