It worked on five hundred documents and broke at fifty thousand

If retrieval was good at five hundred documents and is unreliable at fifty thousand, the most likely answer is that nothing broke. Top-k is a fixed budget in a contest whose entrant count grew a hundredfold. The chunk that used to win still scores exactly what it always scored; it now has vastly more company above it, and the size of the shortlist never changed to compensate.

Put numbers on it. Five hundred documents chunked at roughly eight chunks each is about 4,000 candidates, so a k of 5 is keeping the top 0.13 percent of the corpus. Fifty thousand documents is about 400,000 candidates, and the same k of 5 is now keeping the top 0.00125 percent. That is a hundredfold tightening of the selection criterion, applied silently, by a config value nobody edited.

Prove it with a recall probe before you change anything

The single most expensive mistake here is swapping an embedding model on a hunch. A recall probe costs an afternoon and tells you which half of the pipeline is at fault, so run it first. It measures one thing: for a question whose answer you already know, does the answer-bearing chunk appear in the candidate set at all?

1. Assemble twenty questions whose answer document you can name. Take them from real logs, not from imagination — invented questions use the document's own vocabulary and the probe passes for the wrong reason.
2. Record the expected chunk id for each, by hand. This is the tedious step and there is no substitute for it.
3. Run each question against the full index with k raised to 50, and record the rank at which the expected chunk appears, or 'absent' if it never does.
4. Re-run the same twenty against a restricted index that holds only the original document set — a metadata filter on source or ingest date is usually enough to reconstruct it without a second index.
5. Compute recall at 5 and recall at 50 for both runs. Four numbers. That table is the whole diagnosis.
6. Where the rank moved but stayed inside 50, note the score of the expected chunk and the score of the chunk that beat it. The gap between those two scores is the second most useful number on the page.

Absent from the candidates, or present and ranked twelfth

Everything downstream depends on this split, and teams routinely spend weeks on the wrong side of it. The two branches share a symptom and share no remedy.

If the third row is where you land, retrieval succeeded and the fault is in what happened afterwards; the right passages and a wrong answer is the page that owns that case. If only one named document misbehaves while the corpus as a whole is fine, this page is the wrong one and the single-document forensic sequence is the right one.

The four things a bigger corpus actually changes

Growth does not degrade retrieval in one undifferentiated way. It does four distinct things, each with its own signature in the probe data, and the fix for one is inert against the others.

Candidate crowding has a close relative worth knowing about: because k is zero-sum, a single new source that ranks moderately well on many queries will evict previously winning passages across the board. That is a new source pushing good passages out, and it is the same arithmetic arriving in one step instead of gradually. Neighbourhood density has one too — when the spread is that tight, the same question returning different passages every run stops being a bug report and starts being a measurement of how little signal separates the top of your list.

A shortlist of five was a generous filter at four thousand chunks and is a starvation ration at four hundred thousand. The number did not become wrong; the corpus moved out from under it.

What to change, and in what order

The order matters because each step is cheaper than the next and because a later step is uninterpretable until the earlier ones are settled.

1. Raise k to 50 and re-measure. If recall at 50 is high and recall at 5 is poor, you have an ordering problem and you now know it rather than believe it.
2. Add a reranking pass over the widened list, not over the original five. What a reranker actually does is reorder an existing candidate list, so it pays only when the first stage is already finding the passage — the trade-off is laid out in more candidates versus reranking.
3. If recall at 50 is also poor, stop tuning ranking. The passage is unreachable, and the usual cause is vocabulary: a query built from words the document does not contain. Add a lexical arm as set out in keyword matching versus vector similarity.
4. Extract the two or three fields that scope your corpus — effective date, market, product line — and apply them as metadata filters inside the retrieval query rather than as a post-filter over results.
5. Collapse near-duplicate boilerplate at index time, so one canonical copy competes instead of forty.
6. Only now consider the index itself. Whether the vectors live in the database you already run or in a dedicated store is an operational decision covered in database extension versus dedicated vector store, and it is almost never the cause of a quality regression.

What widening k costs, and where the cost shows up

Going from 5 candidates to 50 barely moves search latency — an approximate index is doing nearly the same graph traversal either way. It multiplies what you send to the model. If all 50 passages reach the prompt, you have multiplied input tokens per answer by roughly ten while traffic stayed flat, which is exactly the shape described in a bill that jumped without more traffic.

The standard resolution is to widen the candidate stage and keep the prompt narrow: fetch 50, rerank, pass 6. Recall is set by the wide stage and prompt size is set by the narrow one. It also avoids stuffing a long context with marginal passages, which brings its own well-documented positional problems.

Where the fix is not

Three things get blamed for this and rarely deserve it. The embedding model, ruled out in ten minutes by comparing the answer chunk's score across the two probe runs. The chunker, unless the probe shows the answer split across a boundary. And the prompt, which cannot influence anything that never reached it.

The durable version of this work is not a tuning session. It is a fifty-line harness that runs the probe set on every index rebuild and fails loudly when recall at 5 drops, which is the kind of narrowly scoped internal tooling we describe under MVP and product builds. The rest of this silo — how passages get chosen, filtered, ordered and cited — sits in retrieval and grounding, inside the wider engineering library.

• retrieval
• recall
• top-k
• scaling