Facts in the middle of a long context get missed

Take one question you know the system answers unreliably. Fix the exact set of passages it retrieves. Then run the same request repeatedly with only one thing changed — the position of the chunk that actually contains the answer, moved from first to last through every slot — and record how often the answer comes out right at each position. That curve is the whole diagnosis. It takes about an hour to produce and it replaces a fortnight of theorising about attention.

Two shapes matter. If the hit rate is high at the start and the end and sags in between, position is doing real work and the repair is to send less. If the hit rate is uniformly poor wherever you put the chunk, position is innocent — the passage does not carry the answer in a form the model can use, which is a chunking or retrieval finding rather than a prompting one.

Running the sweep

1. Pick one question with a single, checkable answer — a number, a date, a named condition. A question whose answer is a paragraph cannot be graded automatically, and a sweep you grade by hand will be run once and never again.
2. Capture the passage set once, as literal text, and store it. From here on retrieval is out of the experiment; you are testing what happens after the passages are chosen.
3. Write the grader before the runs. A substring match on the expected value is usually enough, and a grader written afterwards will be tuned, unconsciously, to the results you already saw.
4. Generate the arrangements. With 8 passages you have 8 arrangements: the answer chunk at slot 1 with the other 7 in their original relative order, then at slot 2, and so on. Keep every other passage in a fixed order so position is the only variable.
5. Run each arrangement 10 times at the lowest temperature the model allows, and record correct or not correct. 10 runs distinguishes 9 out of 10 from 4 out of 10, which is all the resolution this needs.
6. Repeat the entire sweep at a second value of k — 6 passages and 16, say. The interaction between k and position is where the actual finding usually lives.
7. Record the model snapshot identifier next to the results. A sweep spanning a model change has measured two systems, and what a pinned snapshot actually freezes is worth reading before you assume the identifier means what you think.

The sheet you fill in

Five rows is enough for a first pass. Run the full sweep across every slot only if the five-row version shows something worth resolving, because the cost is one model call per run and it multiplies quickly across positions, values of k and repeats.

Reading the shape you get

The four causes the sweep separates

• Too many passages. k was set once, generously, and never revisited. Every passage that could not plausibly hold the answer is a competitor for the model's attention and a line on your token bill, and it earns neither.
• Low-value padding. A fixed k forces the system to send its 12th-best candidate even when only 3 candidates were any good. A relative cutoff — drop anything more than a stated margin behind the top score — sends 3 passages on easy questions and 12 on hard ones.
• The answer split across a boundary. The condition ends one chunk and the exception begins the next, so no single passage answers the question and the model has to join two fragments. This is the failure that generous overlap exists to prevent, at the price described in chunk overlap and what it duplicates.
• Ordering by raw score. Retrieval order is a similarity ranking, not a usefulness ranking, and a fused list from 2 arms is a compromise between 2 orderings rather than an argument about what the reader needs first — which is what rank fusion across 2 result lists is actually doing.

Every passage you send that could not possibly hold the answer is competing with the one that does. There is no neutral filler in a context window.

Ordering by usefulness rather than by score

Once the sweep shows position matters, the ordering becomes a design decision rather than an accident of the ranker. Four moves are worth making, in this order.

1. Deduplicate first. Near-identical chunks occupy 2 slots and contribute one idea, and they are the cheapest slots to reclaim.
2. Apply a relative cutoff. Keep passages within a stated score margin of the best one and drop the rest, so k varies with how clear the retrieval was.
3. Put the strongest evidence where the sweep says the model reads best — usually the ends. If 2 passages carry the answer, one at each end beats both buried together.
4. Label every passage with its source and its effective date in the prompt itself. This does not fix position, and it does give the model something to prefer when 2 passages disagree, which is a distinct failure you will otherwise diagnose as a position problem.

Why sending more is the wrong reflex

The instinct when an answer is missed is to widen k so the evidence is definitely in there. That does raise the chance the answer-bearing passage is present, which is exactly what recall@k measures. It also lowers the chance the passage is used, and the 2 effects run in opposite directions. Somewhere between them is a value of k where total answer quality peaks, and the sweep is how you find it rather than argue about it.

That crossover moves. It moves when you change the reader, when the corpus grows longer chunks, when a new source starts winning slots. Which is the argument for making the sweep a script with a stored fixture rather than a notebook someone ran in April — a small piece of durable internal tooling of exactly the kind we describe under internal tools and operations software. Re-run it on a schedule and on every model change, and the answer to "should we send more context" stops being a matter of opinion.

The neighbouring diagnoses — a passage that never reached the payload at all, and a passage that arrived and was used badly — sit alongside this one in retrieval and grounding, part of the engineering library.

• context
• prompting
• evaluation
• debugging