The nightly sync reported success and half the records are missing

The run is green because green means no exception escaped the process. It is not a claim about completeness, and in most pipelines nothing anywhere makes that claim — no component knows how many records were supposed to arrive, so no component can notice that fewer did. The defect is the absence of a count, and every one of the causes below is invisible until you produce one.

It is expensive because it is quiet and it compounds. A partial night follows a partial night, the gap moves through the corpus as a shifting hole, and the first symptom is a confident answer that omits something — which arrives as a complaint about the assistant, not an alert about the pipeline.

Count in three places, and compare per partition

The confirming check is 3 counts and a join. Do it before forming any theory, because the shape of the divergence names the cause and saves you reading code you did not need to read.

1. Count at the source, using the source's own query rather than the pipeline's. Same predicate the job claims to use, run independently.
2. Count what landed, at the first durable stop after extraction — the staging table, the object store prefix, the raw bucket — before any transformation gets a chance to drop rows.
3. Count what is retrievable, in the index or serving store. This is the number that answers questions, and it is routinely lower than the landed count for reasons nobody has looked at.
4. Do all 3 per partition, never as a single total. Partition by whatever the source is naturally chunked by: day, tenant, prefix, region, source system.
5. Join the 3 counts side by side per partition and sort by the largest absolute gap. Then look at the partitions with a gap of exactly 0 as well, because a run of perfect zeros is usually a filter, not health.
6. Repeat for 14 days of history. A one-night comparison cannot distinguish a chronic loss from last night's incident, and those get fixed differently.

Green because nothing threw

The most common cause is a per-record try block inside a loop, added during a bad week to stop 1 malformed row killing a 6-hour job. It works exactly as intended and it converts a loud failure into a silent one.

• The handler logs and continues. The log line exists, at debug level, in a stream nobody reads, and the job exits 0.
• No counter separates attempted from succeeded. Without both numbers, a 3 percent skip rate looks the same as a clean run from the outside.
• There is no dead-letter destination. The record that failed is gone rather than parked, so even after you find the bug you cannot replay what was lost without a full backfill.
• The exception is often not the pipeline's fault at all — an encoding error, an oversized field, a null where the schema promised none — which is why it deserves a queue rather than a suppression.

Nothing failed. A loop caught 12,000 exceptions, logged each of them at debug level, and exited zero on schedule.

The loop that stopped at the first page

Paginated reads fail silently by design: the API returns a valid, complete-looking response, and the caller decides whether there is more. Get that wrong and the job processes exactly 1 page and reports success.

Amazon S3's ListObjectsV2 is the clearest illustration, because the numbers are documented and fixed. It returns up to 1,000 keys by default; when more exist it sets IsTruncated to true and returns a NextContinuationToken that the caller must send back on the next request. A loop that reads the Contents array and stops sees a normal 200 response, ingests exactly 1,000 objects out of 40,000, and finishes early with no error. The same shape appears wherever continuation is the caller's job: a cursor returned in a response header rather than the body, a page-number parameter that silently caps, or an offset-based read against a table that is still being written to, where rows shift between pages and records fall through the gap.

• The tell is a suspiciously round landed count. Exactly 1,000, 10,000 or 5,000 records is a page size, not a business quantity.
• The second tell is a run duration that stopped scaling. When volume doubles and runtime does not move, the job is not reading everything.
• Prefer cursor or token continuation over offsets against live data, and treat the absence of a next-token as the only legitimate stop condition.
• Assert the loop's own arithmetic: pages fetched multiplied by page size should bracket the record count, and a mismatch should fail the run rather than be logged.

The filter that narrowed without anyone changing it

The third cause is a predicate that was correct when it was written and is not correct now, because the world around it moved. Nothing in the code changed, which is what makes it hard to see in a diff.

The last row is why the third count exists. A document can land perfectly and never become retrievable — an extraction produced no text, a required metadata field was absent, an embedding call failed and was retried into oblivion — and a pipeline that reconciles only source against landing will report full health while the corpus has a hole in it.

Which gap opened tells you where to look

The first row sends people down a wrong path for a week. If all 3 counts agree and records are still missing from answers, the pipeline moved everything it was offered, and the fault is in how change was detected upstream — a different diagnosis entirely.

Make the run assert its own completeness

Every repair above is a bug fix, and bug fixes do not survive the next source, the next schema change or the next well-meaning try block. The durable change is that the run itself refuses to report success without evidence.

1. Emit expected, attempted, succeeded, skipped and failed as counters per partition, from the job itself, and persist them as a run record beside the data.
2. Add a reconciliation step as the last stage of the run, comparing the 3 counts per partition against a stated tolerance — 0 for most transactional sources, a defined small percentage where late arrivals are genuinely normal.
3. Fail the run on a breach rather than warning. A warning in a pipeline is a message to nobody, and a failed run is the only signal that reliably reaches an owner.
4. Route every skipped record to a dead-letter store with the raw payload and the exception, so a fix can be replayed instead of backfilled.
5. Make re-runs idempotent before you turn any of this on, or the first honest failure will produce a duplicate load on top of a partial one.
6. Keep the run record queryable for at least a quarter, because the useful question is rarely about last night — it is which night the gap started.

What to alert on, at what threshold and to whom, is a subject of its own and belongs with the rest of the monitoring design rather than in this diagnosis. The build itself is small and unglamorous: a counter, a comparison and a failure path. Most operators end up assembling it around a managed connector they did not write, which is exactly the middle position described in what assemble means between build and buy — the vendor moves the bytes, and you own the proof that they all arrived.

That ownership is not optional once anything downstream is answering questions autonomously. A retrieval system has no way to know that a partition is missing; it answers from what it has, fluently, which is the failure mode we keep returning to in AI agents in production and the reason completeness checks are part of how we scope AI agents and automation rather than an afterthought.

What count reconciliation will not catch

Counts are a completeness check and nothing else. Three failures pass it cleanly, and each has its own diagnosis. A record can arrive with the right count and the wrong content, because an upstream field was renamed and the pipeline carried on — that is an upstream field changing and nothing breaking. A deletion can travel nowhere while every count stays balanced, which is the record that was deleted and still gets quoted. And a corpus can be complete and old at the same time, which is answers that are a quarter out of date.

Fix completeness first anyway. All 3 of those diagnoses assume the records are present, and none of them can be run honestly on a corpus with holes in it. This page sits in data readiness, pipelines and keeping the corpus true, part of the engineering library.

• pipelines
• data quality
• monitoring
• diagnosis