Multi-hop questions: the shape single-shot retrieval cannot serve

A multi-hop question is one whose answer needs 2 or more retrievals, where a later search cannot be written until an earlier one returns. The dependency is the definition. A question can be long, technical and full of constraints and still be single-hop, because one passage settles it.

Compare 2 questions about the same contract. What is the notice period in the Meridian supply agreement is single-hop: it names the document, and one passage answers it. What is the notice period in our largest supplier's agreement is multi-hop, because you cannot search for it until something tells you who the largest supplier is. The second is not harder to read, it is structurally different.

The test: try to write the search

Write down the query you would type into a search box. If every term in it comes from the question itself, the question is single-hop. If any term has to be filled from another document, it is multi-hop, and the number of blanks is the number of extra hops.

This matters because the 2 failures look identical from outside — a confidently wrong answer — and have opposite repairs. A failing single-hop question is a retrieval quality problem: chunking, vocabulary, ranking. A failing multi-hop question was never going to succeed, and the index is the wrong stage to tune.

Three shapes, only one of which is strictly dependent

Only the first row is dependent in the strict sense. The other 2 belong with it because they break the same design for a related reason: a fixed retrieval budget is split between subjects, so 10 passages become 5 per contract, or 8 about the loudest and nothing about the other 30. Aggregation is harshest and least helped by more hops — counting and list-all questions.

Why one embedding lands between both documents and matches neither

A vector search turns the question into a single point and returns its nearest neighbours. A question naming 2 subjects produces a point pulled toward both, sitting between them — and the passages nearest that midpoint mention both without answering anything: index pages, cover sheets, an email listing both contracts.

So the list looks reasonable and holds no evidence. Adding candidates does not reliably help: rank order near the midpoint is dominated by documents whose topic is the pair rather than either subject. The keyword arm often does better, because an exact name match pulls the specific document up wherever the centroid landed — an asymmetry in lexical and semantic retrieval failing differently.

A long question is not a hard one. A question with a blank only another document can fill is a different kind of question.

The four ways teams handle it

• Decompose in advance. Split into sub-questions, run them in sequence, pass each result into the next. Predictable, and it needs the decomposition right first time.
• Retrieve in a loop. Let the system search, read, and decide whether it can answer or needs another search. More capable and harder to bound, which is why it needs a decision rather than a default.
• Turn the first hop into a lookup. Often hop 1 is not a document search at all: the largest supplier is a query against a table, and a join answers it exactly rather than approximately — a practical point in keeping vectors in the database you already run or standing up a dedicated store.
• Put the whole set in the prompt. Where the corpus is small and bounded, skipping retrieval sidesteps the problem — the conditions for that are in pasting the whole corpus into the prompt or retrieving from it.

Whichever you choose, a multi-hop answer is a chain, and a chain needs per-hop visibility: the query at each step, what came back, where it stopped. Without it every failure arrives as the assistant got it wrong, with no way to say which hop failed. Whether that instrumentation is built on telemetry you already run or bought has a real threshold — building on your telemetry or buying an AI tracing tool.

The architectural question — one retrieval or a loop — is decided in should the system search once, or keep searching, and the worked example is a question that spans two contracts. Deciding which shapes a first version supports, and saying plainly that it does not do the others, is scoping rather than engineering — a boundary we set early in MVP and product builds. Related terms sit in retrieval and grounding, part of the engineering library.

• multi-hop
• retrieval
• question types
• definitions