Library// topic
Data readiness, pipelines and keeping the corpus true
In short
The unglamorous half of every AI build — whether the data is usable at all, how it gets in, how it stays fresh, and how deletions, duplicates and schema drift are handled before anything is indexed or inferred over; one-off migrations of a legacy dataset belong to the integration cluster.
16 pages
definitions
- A freshness SLA is a promise about the worst case, not the averageFreshness is the age of the data behind an answer when it is served, held under a stated ceiling. Latency is how fast a run finishes, and the two can disagree by a week.definition6 min
- Backfill: the word that hides four different jobsFirst historical load, gap repair, transform-change reprocessing and full rebuild are all called backfill. They share a shape and nothing else, including risk.definition6 min
- Change data capture, applied to documents rather than rowsChange data capture makes downstream work proportional to what changed rather than to the size of the corpus. Documents make it hard, because they have no equivalent of a database write log.definition6 min
- Idempotent ingestion: running the same file twice must change nothingIdempotence is a property of the write path decided by the key. Path, arrival order and generated identifiers all look stable and fail on the second run.definition6 min
- Lineage: tracing one answer back to the record and the run that produced itLineage is three identifiers carried on every derived record: which record it came from, which version of the transform produced it, and which run wrote it.definition6 min
- The golden record: which version of the truth the system is allowed to useA golden record is a per-field precedence policy, not a product and not a dataset. Buying a tool implements the rule; it never decides it.definition6 min
diagnostics
- A subset of the PDFs came through as gibberish and nobody lookedA chunk of mojibake embeds happily, indexes happily and retrieves for nothing. No stage errors, so the only defence is a screen that reads the text before it is indexed.diagnostic10 min
- An upstream field changed and the pipeline carried on regardlessA renamed source field does not raise an error. It returns nothing, coalesces to an empty string, and quietly hollows out every record ingested since — until someone plots completeness by day.diagnostic10 min
- Someone updated the document and the system still serves the old textThe indexing job is rarely the culprit. A timestamp that never moved, a hash over the wrong bytes, or an edit the extractor discards all leave the pipeline correctly deciding there is nothing to do.diagnostic11 min
- The figures in the answer are a quarter old and nobody noticedA vague sense that the data feels old becomes actionable the moment you measure it: sample what was actually served, subtract source-modified from indexed-at, and read the distribution.diagnostic10 min
- The ingest queue backs up every Monday morningA backlog that arrives on the same weekday is not a bug. The source hands over a week of work in a few hours, and a pipeline sized for the weekly average cannot absorb it.diagnostic11 min
- The nightly sync reported success and half the records are missingA run that reports success is only claiming that nothing threw an exception. Comparing source, landed and indexed counts per partition is what turns that claim into evidence.diagnostic11 min
- The record was deleted and the assistant still quotes itDeletion is the operation pipelines never test, so it is the one that stops halfway in silence. One tracer record, queried against every store in order, names the hop that ate it.diagnostic10 min
- The reindex has been running for three days and is still not donePercent complete cannot tell a slow job from a stuck one. Committed records per minute can, and the curve's shape names the cause before you open a single log file.diagnostic11 min
- The same customer appears three times under three spellingsTotals that disagree and an assistant that says a customer has two accounts are the same defect: nothing in the chain ever decided that two records describe one company.diagnostic11 min
- Two copies of the same document, and the assistant believes bothAn exact duplicate is harmless. A near-duplicate agrees with the original everywhere except the one clause that changed, and both copies rank convincingly.diagnostic11 min
Other topics in Library
See allRetrieval and grounding: answering from your own contentEverything that happens at query time — how a passage is chosen, ranked, filtered by permission and cited — and how you tell 'the wrong documents came back' apart from 'the right documents, badly used'; the plumbing that gets content into the index lives in the data cluster, and eval methodology lives in the release-gate cluster.20 pagesChoosing the shape of the systemThe decision made before any code exists: what problem shape this actually is, whether it needs an agent loop, a fixed pipeline, a classifier or no model at all, how much autonomy the task can carry, and whether prompting, retrieval or training is the right lever — which process to automate first is an organisational question and sits in the adoption cluster.2 pagesDeciding what to build, and who builds itThe pre-engagement decision — whether to build, buy or configure, whether to hire in-house or bring in a partner, how studios, consultancies, agencies and freelancers actually differ, and how to justify the spend and run diligence on a vendor whose security posture rests on controls rather than certificates.6 pages
Working on something in this space?
Tell us where you are in a sentence or two. We'll tell you honestly whether we're the right team, and what a sensible first slice of the work looks like.
Start the conversation