The right passages were retrieved and the answer is still wrong

If the passage containing the answer was in the context window and the answer is still wrong, retrieval is not your problem and tuning it will not help. The defect is in synthesis: what the model did with evidence it was given. That narrows the repair surface enormously — to the prompt, to the composition of the passage set, or to the question itself — and it rules out all 4 of the index, the embedding model, the chunker and k.

One boundary first, because it decides whether you are on the right page. If the answer contradicts the supplied passages outright and matches what the model would say with no context at all, the passages were not used, and the model answering from memory instead of context is the page you want. This page covers the harder case: the passages were used, and the answer built from them is wrong anyway.

The substitution test, before any theory about the model

The confirming check is embarrassingly low-tech and it is the only one that separates a synthesis defect from an evidence defect. Run it before you rewrite a prompt.

1. Capture the exact passage set that was sent, in the order it was sent, with the character count of each. Not the retrieval results — the payload. Those differ more often than teams expect.
2. Strip out the generated answer and hand a colleague only the question and the passages, with no other context.
3. Give them 10 minutes, and ask them to point at the sentence each part of their answer came from.
4. If they produce the right answer with citations, the evidence was sufficient and the defect is synthesis. Continue to the table below.
5. If they produce the same wrong answer, the evidence was insufficient or misleading and no prompt engineering will save it. That is a coverage problem — go back to what is in the passage set and why.
6. If they cannot answer at all, the question is under-specified or genuinely multi-hop, and the system is being asked for something a single retrieval pass cannot supply.

Four synthesis defects, and what each one leaves in the output

These are distinguishable from the answer text alone, which is what makes them usable. You do not need to know what the model was thinking; you need to know which of 4 shapes the wrong answer has.

The dropped qualifier is the one that costs real money

Conflation is usually caught, because a reader who knows one of the two documents notices the seam. A dropped qualifier is not caught, because what is left is a clean, quotable, entirely plausible sentence — it has simply stopped being true in most cases. A cut-off time that applies only to next-day service becomes the cut-off. A rate that applies to one contract tier becomes the rate. Operational domains are full of these: the gap between a rule and a rule-with-a-condition is the same gap discussed in AI in logistics operations, where nearly every commitment is conditional on service level, lane or equipment type.

The structural cause is usually chunking rather than prompting. If the rule sits at the end of one chunk and its exception at the start of the next, the model receives the rule and never sees the condition, and it is behaving correctly on the evidence it has. Before rewriting instructions, check whether the qualifier was in the payload at all. If it was not, this is a chunk-boundary problem wearing a synthesis costume.

Silent averaging: the answer that appears in no source

Give a model 1 passage saying 30 days and another saying 60 days, ask for the notice period, and the worst outcome is not that it picks wrong. It is 45 days: a number in neither passage, presented with the same confidence as a quoted one, and unfalsifiable by anyone who does not go back to the documents.

This happens because nothing in the request tells the model that disagreement is an allowed answer. Fix it in 3 places rather than 1: state explicitly that conflicting sources must be reported as conflicting, attach effective dates and source names to each passage so recency and authority can settle it, and check on the way out that every number in the answer appears verbatim in some passage. That last check is a regular expression, it adds under 1 millisecond, and it catches the invented midpoint every time.

An answer that splits the difference between two sources is worse than an answer that picks the wrong one, because picking wrong is traceable and splitting the difference is not.

Bridging inference, and the citation that comes with it

The fourth defect is the subtlest: the passages support step 1 and step 3, the model supplies step 2 from general knowledge, and the finished claim reads as though it were sourced. It usually arrives with a citation attached, because the model attaches the nearest plausible source to a claim it produced itself — which is why this defect and a citation pointing at the wrong paragraph are so often the same incident seen from two angles.

The check is to demand a quoted span rather than a document reference. A claim that can name the sentence it came from is grounded; a claim that can only name a document is a hypothesis with a footnote. Requiring the span costs perhaps 40 output tokens per claim and converts an unverifiable answer into one a reader can audit in 30 seconds.

When the passages were right and still not enough

The substitution test sometimes shows that a careful human fails on the same evidence. That is not a synthesis defect, and it has 3 common shapes worth checking before anyone redesigns a prompt.

• The qualifying passage used to be there and no longer is. Because k is a fixed budget, a new source that ranks moderately well across many queries evicts previously winning passages — the mechanism set out in a new source pushing good passages out. The answer degraded because the context got worse, not because the model did.
• The number is in a table and arrived without its header. A value separated from the row and column that give it meaning is worse than a missing value, and the reasons it happens are covered in numbers inside tables never surfacing.
• The question needs a second retrieval that depends on the first. No single-shot passage set can serve it, and the honest response is to route it differently rather than to hope a longer context compensates.

What to change, in the order that keeps the diagnosis intact

1. Add per-claim citation with a quoted span. It fixes nothing on its own and it makes the other 3 defects visible, which is worth more.
2. Add source name and effective date to each passage header in the prompt, so conflicts can be resolved by rule instead of by averaging.
3. Add the output check that every figure in the answer appears verbatim in a passage. Cheap, mechanical, and it catches the defect no reviewer catches.
4. Only then adjust chunk size or overlap, and only where the substitution test showed a qualifier missing from the payload.
5. Re-run the substitution test on the same 20 questions after each change. Synthesis fixes trade against each other — a prompt that forbids inference also refuses questions it should answer — and without a fixed set you will not see the trade.

Most of this is guardrail work rather than model work, and it is the same discipline that makes any autonomous step in a workflow reviewable, which is what we build under AI agents and workflow automation. The neighbouring failures — passages that never arrived, citations that do not resolve, answers that changed without a deploy — sit beside this page in retrieval and grounding, inside the wider engineering library.

• grounding
• synthesis
• answer quality
• evaluation