Search that is never behind.

The problem

Search indexes drift: the application writes the database and then the index, one of them fails, and the index is wrong until the next full rebuild. Rebuilds are slow, and meanwhile users search for things that are not there or find things that are gone.

How it works with Waltail

The four search sinks — Elasticsearch, OpenSearch, Typesense and Meilisearch — turn each change into an upsert or delete keyed by the row's primary key. Batches go through each product's bulk API, and runs of upserts and deletes are flushed in order so a delete never overtakes the update before it. The index is an exact, slightly delayed mirror of the table.

What to watch

• The document is the row's data as-is. Shape the table (or a view-backed table) the way you want the document.
• Create the index or collection with your mapping first if you need more than dynamic mapping.
• Seed the index with a backfill: pick the tables, confirm the estimate, and the current rows arrive through the same sink as read events, interleaved correctly with live changes.

Sinks for this

Questions

More: Webhooks for your database · Cache invalidation · Event-driven services · Audit log and archive.