Every change, kept.

The problem

Audit tables maintained by triggers slow writes, bloat the database and are easy to bypass. Compliance wants a record that lives outside the system it describes.

How it works with Waltail

The S3 sink writes each batch of changes as an NDJSON object under prefix/YYYY/MM/DD/, named by the pipeline and the sequence range it covers. Each line carries the operation, table, new row, previous row when replica identity provides it, sequence and commit LSN. Query it with Athena or DuckDB, retain it by bucket policy, and never touch the production database for an audit question again.

What to watch

• Set REPLICA IDENTITY FULL on audited tables so old holds the full previous row, not just the key.
• Larger batch sizes mean fewer, larger objects.
• Object keys are dated by delivery time; the commit LSN inside each line is the authoritative order.
• Start the archive with a backfill of the tables as they are today, then let the stream append from there.

Sinks for this

Questions

More: Webhooks for your database · Cache invalidation · Search index sync · Event-driven services.