A freshness SLA is a promise about the worst case, not the average
In short
A freshness commitment is a ceiling on how old the data behind an answer may be, measured at the moment the answer is served and met for a stated share of answers. It is not pipeline latency. A pipeline whose every job finishes in 6 minutes can still serve week-old figures, because the cadence of the slowest stage sets the age, not the speed of any single run.
Key takeaways
- Age runs from the source change to the moment of serving, not from when the pipeline saw it.
- Latency measures a run. Freshness measures an answer. A fast pipeline can serve week-old data.
- Cadence dominates: the slowest scheduled stage sets the worst case, whatever the runtimes are.
- Commit to a percentile and a hard ceiling. In batch systems the mean describes nobody.
- A ceiling nobody is rostered to repair is a target, not a commitment. State the hours.
Freshness is the age of the data behind an answer at the instant it is served, expressed as a ceiling that holds for a stated share of answers over a stated window. "No answer draws on source data more than 24 hours old, for 99 percent of answers, measured monthly" is a commitment. "The sync runs nightly" is a schedule.
The measurement point makes it useful and makes it hard. Age counts from the moment the change happened at source, not from when your pipeline noticed it. Measuring from ingestion reports your own promptness and says nothing about what a reader cares about.
Every job finishes in minutes and the figures are a week old
Latency is the time a change takes to cross one stage; freshness is the accumulated age at the end. They come apart because most of the age in a batch chain is waiting, and waiting appears in no job's duration metric.
| Stage | Cadence | Run duration | Worst-case wait added |
|---|---|---|---|
| Source export | Weekly, Sunday 23:00 | 18 minutes | 7 days |
| File arrival and validation | Every 15 minutes | 2 minutes | 15 minutes |
| Parse, chunk and index | Hourly | 6 minutes | 1 hour |
| Answer cache | Expires after 6 hours | n/a | 6 hours |
Total run time across that chain is 26 minutes; worst-case age is a little over 7 days and 7 hours. Every dashboard is green, every job is fast, and a reader can be handed a figure that changed 8 days ago. The fix is the top row, not a faster parser.
Why the number has to be a percentile with a ceiling behind it
In a batch system the age distribution is not a bell curve. Just after a run everything is young, just before the next one everything is old, and the mean sits where almost no record ever is. The mean age of a corpus refreshed every 24 hours is roughly 12 hours, true of nothing anyone is served.
- Report percentiles, not means. The p50 describes the middle of the cycle, the p95 what an unlucky reader gets, and the gap is the length of your slowest interval.
- Commit at the percentile with a hard maximum behind it. "p95 under 24 hours" and no ceiling permits a tail of 6-month-old records, which is the tail that causes the incident.
- Segment by source. One corpus with 5 feeds has 5 age distributions, and the aggregate hides the source that fell over 3 weeks ago.
What committing to a ceiling obliges you to build
A freshness figure in a specification is cheap to say and not cheap to keep. Four things must exist before the sentence is honest.
- A source-change timestamp that survives the chain, carried untouched through every transform, merge and reprocess — the same plumbing as tracing an answer back to the record and the run that produced it.
- A measurement taken at serving time, logged per answer. Age computed on the index misses the answer cache, usually the largest single contributor.
- An alert on the percentile, not on job failure. A pipeline succeeding nightly against a source that stopped exporting raises nothing while the percentile climbs.
- A named owner and stated hours. A ceiling nobody is rostered to defend is a target with a nice name. Ours is business hours, Monday to Friday, and saying so beats implying otherwise.
Two operations break the measurement if nobody defends it. A reprocess that stamps records with the time they were reprocessed makes a whole corpus look newly fresh while nothing underneath changed — one reason to separate the kinds of reprocessing, in the four jobs the word backfill hides. A replay that writes duplicates with new timestamps corrupts the distribution, an argument for running the same file twice and changing nothing.
Check what you already have first. Most synchronisation products report their own transfer latency on a dashboard and teams read it as freshness; it is one hop, silent about the export cadence above it and the cache below it. Working out what an incumbent covers is we already pay for a tool that almost does it. The age panel itself — percentiles per source, oldest record, last arrival — is small internal tooling and operations software.
One distinction to keep straight: an objective is an internal target you manage to, an agreement is a commitment with a consequence somebody signed. Check which you are being asked for before quoting a figure. Choosing the figure is how out of date an answer may be, within data readiness and pipelines in the engineering library.
Frequently asked questions
Short answers to the follow-ups this page tends to raise.
What is the difference between data freshness and pipeline latency?
Latency is how long one stage takes to move a change; freshness is how old the data behind a served answer is. They diverge because most of the age in a batch chain is waiting between runs, which no job duration records. A chain whose stages run in 26 minutes in total can still serve data over 7 days old if the source exports weekly.
How do you measure data freshness end to end?
Stamp each record with the time the change happened at source, carry that field through every transform, and at serving time subtract it from the current time for each passage the answer used. Log that age per answer and report percentiles over a window. Measuring from when your pipeline first saw the record reports your own promptness rather than the age of what the reader received.
Should a freshness target be an average or a percentile?
A percentile, with a hard maximum stated alongside it. Batch refresh produces an age distribution where everything is young just after a run and old just before the next, so the mean lands at a value almost no served record has. "p95 under 24 hours, ceiling 48 hours" describes the reader's experience; an average of 12 hours describes the schedule.
Does committing to a freshness number require monitoring?
Yes, plus an owner and stated hours of cover. A pipeline can succeed every night against a source that stopped sending data weeks ago, so job-success alerts never catch a freshness breach; only an alert on the age percentile does.
- freshness
- measurement
- definitions
- pipelines
The work behind this page
Builds from our portfolio that this page draws on.
ShipSight
A supply-chain control tower that tracks every shipment across ocean, air and ground, predicts each ETA with a confidence score, and flags at-risk shipments before they slip.
LogisticsAskVault
An AI internal knowledge-search platform that answers employee questions from your own docs — grounded in citations, with knowledge gaps surfaced and deflection tracked.
Productivity AIRead next
- Lineage: tracing one answer back to the record and the run that produced itLineage is three identifiers carried on every derived record: which record it came from, which version of the transform produced it, and which run wrote it.definition
- Idempotent ingestion: running the same file twice must change nothingIdempotence is a property of the write path decided by the key. Path, arrival order and generated identifiers all look stable and fail on the second run.definition
- Backfill: the word that hides four different jobsFirst historical load, gap repair, transform-change reprocessing and full rebuild are all called backfill. They share a shape and nothing else, including risk.definition
- The figures in the answer are a quarter old and nobody noticedA vague sense that the data feels old becomes actionable the moment you measure it: sample what was actually served, subtract source-modified from indexed-at, and read the distribution.diagnostic
- A subset of the PDFs came through as gibberish and nobody lookedA chunk of mojibake embeds happily, indexes happily and retrieves for nothing. No stage errors, so the only defence is a screen that reads the text before it is indexed.diagnostic
- An upstream field changed and the pipeline carried on regardlessA renamed source field does not raise an error. It returns nothing, coalesces to an empty string, and quietly hollows out every record ingested since — until someone plots completeness by day.diagnostic
Working on something in this space?
Tell us where you are in a sentence or two. We'll tell you honestly whether we're the right team, and what a sensible first slice of the work looks like.
Start the conversation