An upstream field changed and the pipeline carried on regardless

The vendor shipped a release on a Tuesday. Somewhere in it, a field was renamed. Your connector asked for the old name, got nothing back, wrote an empty string, and every stage after that behaved impeccably: the record parsed, the chunk embedded, the index accepted it, the assistant answered. Six weeks later somebody notices the answers have stopped mentioning who owns anything.

There is no error to find because no rule was broken. Pipelines are built to be forgiving — a missing key returns a default, a validator with no declared constraints accepts anything, a text field takes whatever it is given. Forgiveness is the right default for surviving a messy source and the wrong default for noticing the source changed. So the diagnosis is a measurement, not a search through logs.

Two numbers per field, plotted against ingest date

Completeness measured over the whole corpus tells you nothing: an old, healthy majority drowns a recent, broken minority. Measured per day of ingest, the same data draws a cliff with a date on it.

1. Work from the retained raw payloads, not the transformed table. The transform is where the evidence was destroyed, so anything computed after it will look consistent by construction.
2. For each field, compute 3 figures per ingest day: the share of records where it is null, absent or an empty string; the count of distinct values; and the mean character length.
3. Add a fourth that almost nobody collects: the number of keys present in the payload that your mapping does not know about, per day. A new key appearing on the same day an old one empties is the rename, proven.
4. Plot each field by day and look only for steps. Drift is discrete because a release is discrete; a gradual decline is a different problem, usually a growing subpopulation that was always incomplete.
5. Take 1 record from the day before the step and 1 from the day after, and diff the raw payloads. This takes 5 minutes and settles arguments that otherwise run for a week.
6. Check the queue before you blame the source. A step that starts on a Monday can be a backlog artefact rather than a schema change, which is the pattern in the ingest queue that backs up every Monday.

Five ways a source changes shape, and the fingerprint each leaves

The second and third rows are the ones that reach an answer looking confident. A widened type or a remapped category leaves text present in the chunk, so retrieval succeeds and the sentence reads normally; only the meaning is wrong. Two knock-on effects are worth checking at the same time. A source that starts sending a display name where it used to send a code manufactures new spellings of entities you thought were resolved — one customer under three spellings arriving by a new route. And a full re-export after a schema change often lands as a second copy of every document rather than an update, producing the near-duplicates in two copies and one contradiction.

Why every layer accepted it without complaint

Each individual permissiveness is defensible. Stacked, they guarantee that a schema change becomes a quality problem rather than an outage.

• The read is optional by default. A dictionary lookup with a fallback, a null-safe accessor, a template that renders nothing for a missing value: all return successfully.
• The validator was never told to be strict. JSON Schema allows additional properties unless additionalProperties is set to false, so a payload carrying brand new keys validates cleanly against a schema that has never heard of them.
• Presence and emptiness are conflated. A required-field rule that only checks for the key's existence passes an empty string, and an empty string survives every subsequent stage as legitimate content.
• The mapping only names the fields you use. Anything outside that list is invisible by design, which is exactly why a new key arriving is worth counting explicitly.
• Text stages accept anything. Chunking, embedding and indexing have no opinion about semantics, so a stringified object is processed as faithfully as a paragraph.

Forgiving parsers are the right answer for a messy source and the wrong answer for noticing the source changed. Both are true at once.

Put a contract at the boundary and fail the batch, not the field

The repair is a check at the point of arrival, before any transform runs, that describes what the source is expected to look like. It does not need to describe everything — 6 to 10 assertions about the fields the build actually depends on will catch the overwhelming majority of drift.

• Assert presence and non-emptiness separately, because a key with an empty value is the exact failure this class of incident is made of.
• Assert types and, where it matters, formats. A date that is expected to parse should be parsed at the boundary rather than 4 stages later inside a template.
• Assert closed value sets for anything you map. An unrecognised status is information, and defaulting it to the most common value throws that information away.
• Assert that no unknown keys are present, or at minimum count them and alert. This is the only check that fires on the day of the change rather than the day someone complains.
• Quarantine the failing batch rather than dropping the field. Silently continuing with a missing value is the behaviour that caused the incident; the alternative is a held batch with a diff attached and somebody named to look at it.

Fail the batch, not the record, where the change is structural: 100,000 records failing the same assertion on the same day is one event, and 100,000 individual errors is a pager nobody reads. The alerting philosophy that goes with this — thresholds on absence, volume and shape rather than exceptions — is set out in what to alert on in an ingest pipeline.

None of this is exotic engineering. A validation step, a quarantine table and a small screen showing held batches with the offending diff is a few days of work, and it belongs in the first version rather than the hardening phase — the kind of thing we scope into MVP and product builds instead of leaving for the incident that justifies it.

Repairing the window that already went through

Fixing the mapping stops the bleeding and repairs nothing. Every record ingested between the change date and the fix is still in the index with a hollow field, and those records are indistinguishable from correct ones to everything downstream.

1. Bound the window precisely. The step in the plot gives a start date; the deployment of the fixed mapping gives an end date. Reprocessing more than that window wastes days for nothing.
2. Reprocess from retained raw payloads, not from a fresh export. A new export reflects the source as it is today, which silently mixes a repair with an update and makes the result unverifiable.
3. Verify with the same plot. The null rate for the repaired field across the affected days should return to its pre-change level. If it lands somewhere in between, the mapping is still wrong.
4. Check for orphans left by the first pass — chunks written under the broken mapping that the repair did not overwrite.

This is a gap repair rather than a rebuild, and the distinction matters for how it is planned and how risky it is — the 4 different jobs people call a backfill are separated in what a backfill really covers, and the staged execution for doing it against a live system is running a backfill without taking the system down.

The part this page deliberately stops short of

You now have a date, a field, a fingerprint and a repair window. What this page does not trace is how a hollow field becomes a confidently wrong sentence — the specific hop where a missing value stops being an absence and becomes fluent prose is followed end to end in how schema drift reaches an answer.

It also leaves the commercial question open. When the source is a vendor product that changes its export without notice, the engineering fix is a contract check and the real question is what that coupling costs you over time, which is the argument in pricing the cost of leaving before you arrive. Both sit in data readiness and pipelines, part of the engineering library.

• schema drift
• data contracts
• ingestion
• diagnosis