Surrogate keys: purging one story without flushing the front page
In short
A surrogate key is a label attached to a cached response at render time so the response can later be invalidated by name. The publisher's problem is not the header but the key set: one story appears on the article page, the section index, the front page, a tag page, a feed and the app, and the keys on those responses decide whether a correction reaches all of them.
Key takeaways
- Keys are attached at render time, because only the renderer knows which entities went into a response.
- One story update touches at least 6 surfaces, and each needs a key that reaches it or the correction stops short.
- Purge against the pre-edit key set as well as the post-edit one, or the old section and tag pages keep the old story.
- Header size limits are real and silent: past the cap, keys are dropped rather than rejected, so purges quietly miss.
- A full flush is not a fallback. It arrives as an origin traffic spike at exactly the moment the story is busiest.
A surrogate key is a label a CDN indexes a cached response against, so the response can be invalidated later by name rather than by URL. The origin attaches the keys as a header when it renders the response; a purge names a key, and every response indexed against it is invalidated at once. That is the whole mechanism, and it is not the hard part.
The hard part is the key set. A single story appears on its own page, the section index, the front page, one or more tag pages, an RSS or JSON feed and an app endpoint — more if it was quoted in a live blog or a newsletter archive. Whether a correction propagates in seconds or sits on the front page for an hour is decided entirely by which keys were attached to those responses when they were rendered.
The key set for one story
Write it out for one article: every surface, the cached object behind it, and the key that reaches it. A row with no key serves stale content until its own time-to-live expires, and nobody will connect the two.
| Surface | Cached object | Key it must carry | Consequence when the key is missing |
|---|---|---|---|
| Article page | The rendered HTML for that URL | story:1234 | The correction is invisible on the story itself, which is the failure everybody notices |
| Section index | A paginated list page | story:1234 and section:politics | The headline in the list disagrees with the headline on the story |
| Front page | The highest-traffic object you own | story:1234 plus a key for the curation object that placed it | Either the promo stays wrong, or somebody flushes the whole front page to fix one line |
| Tag or topic page | A list keyed on a tag that can change during an edit | story:1234 and tag:<each tag, before and after the edit> | Removing a tag leaves the story on the old tag page indefinitely |
| Feeds and syndication | RSS, JSON, partner endpoints | story:1234 and feed:<name> | Aggregators and partners keep redistributing the uncorrected version |
| App and API responses | JSON for the mobile clients | story:1234 and the collection keys the app reads | The web is right and the app is wrong, which reads to the desk as a mystery |
Two entity types generate the rows nobody thinks of. Embedded content — a quoted story card, an author block, a data table — makes the containing response depend on entities other than the one being edited, so it needs their keys too. Curation objects mean the front page can change with no story changing at all, so the placement needs its own key rather than borrowing the story's. Where placement comes from a planning artefact, the keys follow the story budget held as a data object.
Attach the keys at render, never at purge
The rule that makes this work: whatever assembles a response declares what went into it. At purge time nobody knows which of last week's cached pages embedded story 1234 — that knowledge existed only in the renderer, at the moment it fetched the entities. So the renderer emits the key header listing every entity it touched, and a purge only ever names the entity that changed.
This is also the strongest practical argument for holding a story body as structured blocks rather than as a markup blob. A block list can be walked to enumerate the embeds it contains; a blob has to be parsed with a regular expression and hoped over, which is why the block body versus markup blob decision reaches further than it first appears.
Where key sets go wrong
- Purging only the post-edit key set. A story that moves from Business to Politics, or loses a tag, still sits in the old section and tag pages. The publish event has to carry both the before and the after key sets, or the old surfaces keep the old copy until their time-to-live expires.
- Keys for the story but not for its dependencies. Update a reporter's name or a topic page's description and every response that embedded it is stale, because the invalidation was scoped to articles.
- Keys generated from display data. A key built from a slug breaks the moment the slug is edited, which is exactly when you most need the purge to work. Build keys from immutable identifiers.
- Sitemaps and feeds left out of the set. Both are cached objects like any other. A news sitemap that sits behind a long time-to-live is stale in precisely the window it exists to serve, which is one of the constraints behind the news sitemap and its two-day window.
- No verification after the purge. A purge call returning success means the request was accepted. Whether every point of presence dropped its copy is a separate question, and the only honest answer comes from re-fetching.
A full flush is not a fallback plan. It is a decision to send every reader to the origin at the moment the story is busiest.
The instinct behind a full flush is natural: a correction has to be live now, the key set is untrusted, so somebody clears everything. The cache empties, the origin takes the front-page load cold, and the incident stops being about a headline. Precise invalidation is what keeps the correction path usable during a big story, and the same design makes some readers still seeing the old version a diagnosable problem rather than a recurring mystery.
None of this is expensive to build; it is expensive to retrofit, because it means changing every renderer to declare its dependencies. That makes it first-weeks work in a platform build rather than last-weeks, which is how we scope it in MVP and product builds. The rest of this silo sits under CMS and publishing platform engineering, and the wider platform work is in our media and publishing practice.
Frequently asked questions
Short answers to the follow-ups this page tends to raise.
What is a surrogate cache key?
It is a label attached to a cached response by the origin, usually as a response header, that a CDN indexes the response against so it can be purged by name later. One response can carry many keys, and one key can be shared by thousands of responses. Purging the key invalidates every response holding it, which is how one article update clears its own page, the section index and the front page without touching anything else.
How is a surrogate key different from purging by URL?
Purging by URL requires you to know every URL the content appears on, and for a publisher that list is unknowable — paginated indexes, tag pages, feeds and app endpoints all change as content is curated. A key inverts the problem: the renderer records what went into each response, and the purge names only the thing that changed.
How many keys should one response carry?
As few as express the response's real dependencies, because the header has a hard size cap and overflow is silent. In practice that is one key per entity the response renders, plus collection keys for the lists it belongs to — not a key per field, template or variant. On list pages, cap item keys deliberately and let the list carry a collection key instead.
Should the CMS purge directly, or emit an event?
Emit an event and let a service subscribe to it. Purging directly from the editing application couples the newsroom's save button to a network call that can time out, and leaves no record of what was invalidated. An event stream gives you retries, an audit trail, and one place to add the pre-edit key set — the part teams discover they need only after a story has moved section.
- caching
- CDN
- invalidation
- publishing
The work behind this page
Builds from our portfolio that this page draws on.
Read next
- Some readers still see the old version of a corrected storyA correction that is right for you and wrong for a colleague is held by exactly one layer. Ten minutes of evidence tells you which, and stops the reflex to flush everything.diagnostic
- News sitemaps: the two-day window, and what the archive stays out ofA news sitemap is a rolling window, not an index. At publisher scale the failure is inclusion — archives, republished timestamps and paginated lists leaking into a file meant for two days of news.definition
Working on something in this space?
Tell us where you are in a sentence or two. We'll tell you honestly whether we're the right team, and what a sensible first slice of the work looks like.
Start the conversation