The assistant quoted a policy that was replaced last quarter

Before changing anything, find out whether the old passage is still reachable. Take a sentence that appears only in the superseded version — a phrase the replacement rewrote — and run it as a query. Then look at the candidate set the retriever returns, not the final answer. That one observation splits the problem into three, and the three have different owners and no overlapping fixes.

The reason to start here is that all three failures produce the same complaint, and two of them are invisible from the answer alone.

The five-minute probe

1. Get the trace for the exact answer that was reported. Not a reproduction — the original request, with its retrieved passages, filters and timestamps. If you cannot get from a complaint to its trace in one step, fix that first: complaint to trace in one hop.
2. Pick a probe phrase that exists only in the superseded text. 6 to 10 words that the replacement rewrote. If the two versions differ only in numbers, use the surrounding sentence instead, because an identifier alone retrieves badly.
3. Query it and inspect the raw candidate set. Before reranking, before truncation to the prompt. You are asking one question: is the superseded chunk in there at all?
4. Run the same probe with the supersession filter forced on and forced off. If the results are identical, the filter is not doing anything — which is a more common finding than a broken filter.
5. Check what the citation actually points at. An answer can quote current text and cite the wrong location, which reads to a user as an outdated answer; that is a different fault, covered in when the citation points at the wrong paragraph.

That last row happens more often than people expect, and it is a different investigation: the replacement document is not being found either, and the model is answering from whatever was nearest. The forensic sequence for a document that will not come back is in you know the document is indexed and it still never comes back.

Ranked causes at query time

Ordered by how often each turns out to be the cause when the probe shows the old chunk is reachable.

Why the old version out-ranks the new one

When both versions are live and the superseded one keeps winning, the usual explanation is structural rather than mysterious. A policy that has been in force for years accumulates length, examples, appendices and internal cross-references. Its replacement is often shorter and cleaner. More text means more chunks, more chances to match, and a higher density of the exact vocabulary a user types.

• Chunk count asymmetry. An old document producing 40 chunks against a replacement producing 12 has more than 3 times the tickets in the same lottery.
• Vocabulary drift. Users search with the words they learned from the old document, which are literally the words it contains — and which a lexical scorer such as BM25 rewards directly.
• Candidate crowding. At a fixed top-k of 8, more near-duplicates from one source push everything else out, an effect that grows with the corpus and is analysed in retrieval degrades as the corpus grows.
• Reranker inheritance. A reranker can only reorder the candidates it is given, so if the old version dominates the first stage, a second scoring pass mostly confirms it.

A superseded document does not need to be more relevant to win. It only needs to be longer, older and written in the vocabulary your users learned.

The decision tree

One branch, one owner, one change. Resist fixing two layers at once — you will not know which one worked.

1. Probe returns the old chunk, and the chunk has no status or effective-date field. Owner: the team that defines the chunk schema. Change: carry document status and effective dates onto every chunk, so that a filter has something to act on at query time.
2. Probe returns the old chunk, the field exists, the filter does nothing. Owner: whoever owns the query builder. Change: add the predicate, and confirm it executes inside the vector search rather than as a post-filter on the results.
3. Probe is clean, live answers still quote the old text. Owner: the serving path. Change: identify which cache — a Redis answer cache, an embedding cache, the HTTP layer or the client — and tie its invalidation to the document lifecycle rather than to a 24-hour timer.
4. Both versions returned, old one first, and both are meant to be live. Owner: the product decision-maker, not engineering. Change: decide whether historical versions are answerable at all, and if so under what phrasing.
5. Everything above is clean and it happens anyway. Owner: retrieval quality broadly. Change: treat it as a recall and ranking investigation rather than a supersession one, and stop patching the filter.

Design rules that stop it recurring

Four rules, all cheap at design time and all expensive to retrofit. They are query-time rules: how the old document leaves the index is a separate concern with its own failure modes.

• Status and dates live on the chunk. A document-level record is not enough, because the filter runs against chunks. 4 fields: effective from, effective to, status, and the identifier of the document that replaced it.
• Answers state the version they used. The effective date belongs in the rendered answer, not only in a citation panel. A user who can see "effective from 1 March" catches a staleness bug no evaluation set anticipated.
• Supersession is filtered, permissions are filtered, recency is ranked. Keep those three in their own lanes. Conflating them is what produces empty result sets and unexplainable orderings.
• Decide the historical-answerability question before launch, not after the first complaint — the decisions to settle up front are enumerated in the version and supersession checklist.

There is a governance point underneath all of this. A retrieval system inherits whatever version discipline the source content already has, and most document stores have none: the current version and three drafts sit in the same folder, distinguished by a filename convention that a human reads correctly and a pipeline does not. No amount of query-time filtering fixes a corpus where supersession was never recorded.

The wider set of query-time faults — recall collapse, permission filtering, citation alignment, ranking — is mapped across retrieval and grounding in the library. If you are building the system rather than debugging one, this class of failure is one of the reasons unattended assistants need containment patterns rather than better prompts, as argued in what actually breaks with AI agents in production, and it is the kind of behaviour we specify explicitly on AI agents and automation work.

• retrieval
• grounding
• versioning
• diagnostics