Change data capture, applied to documents rather than rows

Change data capture is the practice of identifying which records changed since a known point, so that downstream work is proportional to the change rather than to the size of the corpus. Reprocessing 40 documents instead of 400,000 is the point, and the rest of the design follows from how confidently you can name those 40.

The term came from database engineering, where it can be exact. A relational engine writes every committed change to a durable log before treating it as done — PostgreSQL calls that the write-ahead log, MySQL the binary log. Reading it yields every insert, update and delete in commit order, nothing missed and nothing inferred.

What a table row has that a document does not

A shared drive, a mailbox, an SFTP dump or a content system rarely offers anything comparable. Some expose a change feed, many expose a listing with a modification time, plenty expose nothing beyond a scheduled export. Three properties the database log gives away for free go missing.

• Completeness. The log records every commit by construction. A listing records state at the moment you asked, so a change made and undone between 2 polls leaves no trace.
• Ordering. The log is a sequence, so 2 edits to one record arrive in order. A poll returns a snapshot, and a document edited twice shows only the second edit.
• Deletion. A removed row appears in the log as a delete. A removed document simply stops appearing, detectable only by listing everything and comparing — hence making a delete travel all the way to the last cache.

The three places a document change signal comes from

Cost decides between them more often than teams expect: reading every object is fine across 40,000 documents and untenable across 40 million.

What a capture step owes the rest of the pipeline

Capture is not finished when it emits a list of changed keys. Four obligations follow, and dropping any of them produces a failure that surfaces weeks later somewhere else.

1. Emit an identity, not a path. Downstream needs a stable key to update against, or a document that moves folders arrives as a new record.
2. Stamp each change with the run that saw it, so "which run introduced this passage" stays answerable — the argument in tracing an answer back to the record and the run that produced it.
3. Advance the cursor only after downstream work succeeded. A cursor advanced at capture time turns one processing failure into a permanent gap.
4. Schedule a reconciliation sweep. A full listing compared against what you hold, weekly or monthly, is the only part of the design that repairs its own misses.

Three jobs this term gets used to cover and does not

• It is not replication. Copying a changed record into a second store is the step after capture, and a tool doing both is doing 2 things.
• It is not conflict resolution. When 2 systems report conflicting updates to one entity, capture reports both and decides nothing — the precedence policy in the golden record in a messy source set settles it.
• It is not freshness. Detecting an edit within a second and indexing it overnight still serves a day-old answer, which is why the commitment is measured at serving time — what a freshness commitment obliges you to build.

One practical consequence for anyone deciding what to build. Where the source is mainstream and a maintained connector exists, capture is commodity plumbing; where it is a bespoke internal system, nobody else will ever write it and it is core work — the line drawn in splitting a stack into commodity and core, usually the first question on a product build. Both sit in data readiness and pipelines, part of the engineering library.

• change data capture
• ingestion
• definitions
• pipelines