Backfill: the word that hides four different jobs

A backfill is any run that writes records for a period the live pipeline has already passed. That one shape covers 4 operations people describe with the same word, differing in what triggers them, what they rewrite and what they can destroy. Agreeing which one is being asked for is most of the risk management.

The four operations the word is doing duty for

Only the second row should ever be routine. Gaps happen weekly in any pipeline reading systems it does not control, and repairing them should be a button rather than a project. The other 3 are events: each needs a named owner, a written scope, and agreement on what the corpus looks like halfway through.

The two words missing from almost every backfill request

"Can you backfill the last 2 years" is not a specification. Five answers turn it into one, and the fourth is the one that gets skipped and later hurts.

1. Which sources, named individually. A range across 6 feeds is 6 jobs with 6 retention limits, and the shortest one decides the real answer.
2. The time boundaries, inclusive, against a named clock. A range in local time spanning a daylight-saving change is a 1-hour ambiguity somebody finds later.
3. Which layers get rewritten: source records, chunks, embeddings, summaries. Each layer multiplies the cost and the time.
4. What happens to deletions inside the window. A record created in March and deleted in June is resurrected by a naive backfill, which is worse than a missing one.
5. The cut-off, and the rule for records after it, or the backfill and the live pipeline fight over the same rows.

Why reprocessing is the expensive one and rebuilding is the risky one

A transform-change reprocess touches every record whether or not anything upstream moved, so its size is the whole corpus every time. On an embedding-heavy corpus that is real money, and the unit is chunks not documents: a 40-page report can be 120 chunks, so 50,000 documents is millions of paid calls. It is slow too, meeting a rate limit no concurrency gets past.

A full rebuild costs less and risks more. Its purpose is to catch the drift incremental updates accumulate silently — the case in rebuilding the whole index against updating only what changed — and its danger is the window where the index is live and half-populated. In a stream it becomes a replay from a stored offset competing with live consumption, worth knowing before choosing between scheduled batches and a stream.

Two things to check before promising anyone a backfill

The first is whether the history is reachable. The extraction route decides it: a sanctioned export usually offers a full range, a file or email drop may retain only 90 days, and a route driving the source's interface often cannot reach history at all — the ranking in scheduled exports against driving the interface. A source that discarded the period is not a scheduling problem.

The second is whether reprocessing may overwrite. Where a record must be reproducible as it stood on a date — clinical, claims and regulated financial work, as in AI across healthcare operations — a reprocess that replaces the stored text destroys the evidence of what the system said at the time. Version the derived rows and keep the superseded ones addressable.

It is a useful question for a vendor or build partner: what happens when you change the chunking rule on 400,000 documents. A specific answer describes a parallel index, a switchover and a cost; a vague one about automatic reindexing describes a hope. That is the probe in how to test an AI-native claim, and it belongs in the scoping of any AI automation and agent engineering. The rest sits in data readiness and pipelines, part of the engineering library.

• backfill
• reprocessing
• definitions
• pipelines