You know the document is indexed and it still never comes back

A single document that never surfaces has been dropped by exactly one of four layers: extraction, indexing, filtering or scoring. From the outside all four produce the same observation — you ask a question the document answers, and it is not in the results. Inside, they have nothing in common, so guessing costs days. Probe them in cost order and stop at the first failure.

Three things are needed before you start, and assembling them is not optional. The document's identifier in your own store, not the filename. One query it should obviously win. And the specific sentence in it that answers that query, copied out verbatim, because two of the four probes use that sentence directly. If the chunk id you are chasing came from a citation in an earlier answer, confirm the citation itself is sound first — a citation pointing at the wrong paragraph sends you hunting a chunk that was never the right one.

Probe one: does the chunk exist, and does it contain text

Fetch the chunks for that document id directly from the store, bypassing search entirely, and look at two things: how many there are, and how long each one is. This takes 2 minutes and catches the failure that wastes the most time downstream.

• Zero chunks. The document is in your metadata table and not in the index. Ingestion partially failed, and the source of truth for 'is it indexed' was the wrong table.
• One chunk where you expected forty. Extraction returned a fragment — usually a cover page, an error string, or the text layer of a scanned file that has none.
• Chunks under about 40 characters. Page furniture, headers, or whitespace. These embed perfectly happily and are the trap this probe exists to find.
• The right number of chunks, each with plausible text. Extraction and chunking are fine; continue to probe two.

Probe two: does it survive the filter

Run the identical query twice: once as the application runs it, once with every filter stripped off. If the document appears in the unfiltered run and not the filtered one, you are finished — the fault is in the filter and no amount of embedding work will touch it.

• A null in a field the filter tests. A filter on region excludes every document whose region was never populated, which is usually the oldest and most authoritative material.
• A type mismatch on dates. An effective date stored as a string compares lexicographically, so a range filter silently excludes rows whose format differs by a leading zero or a time zone suffix.
• Permission resolution returning an empty set. When the group lookup fails open on error the filter matches nothing, which looks like a retrieval problem and is an authorisation problem.
• A filter applied after the nearest-neighbour search rather than inside it. The search returns k candidates, then the filter eats most of them, so the document was never given a chance to compete.

That last one is the systemic version rather than the single-document version. It degrades every query at once rather than one file, and the measurement that proves it before any redesign is attempted belongs to recall collapsing after permission filtering.

Probe three: is the chunk reachable by its own words

Copy a distinctive span of about 12 words directly out of the chunk and issue it as the query. This is the strongest possible query for that chunk — identical vocabulary, identical phrasing, no paraphrase to bridge. If the chunk does not come back first, it is not in the searchable set your queries actually reach.

1. Pick the span from the middle of the chunk, not the start, and avoid boilerplate. A sentence containing a rare noun is ideal.
2. Run it through the same query path the application uses, including any rewrite step, so you are testing the live route rather than a clean one.
3. If it comes back at rank 1, the chunk is reachable and the problem is that real user phrasing does not get near it. Continue to probe four.
4. If it does not come back at all, run the same span against the store as a raw text match — a SQL LIKE or a BM25 query over the same rows. A hit here with a miss in search means the vector and the text belong to different records — the classic symptom of an interrupted re-index that wrote text rows without replacing vectors.
5. If the span contains an identifier — a part number, a clause reference, a policy code — treat the result with suspicion either way. Identifiers are the one query class where similarity is structurally the wrong instrument, which queries with a part number or a clause reference coming back empty covers in full.
6. Repeat the exact-text probe three times. If the result changes between runs on a static index, stop diagnosing content and look at the search layer instead.

Probe four: where does it rank, and against what

Now run the real user query with k raised to 100 and record two columns: the target chunk's similarity score, and the scores of the five chunks that beat it. The gap between them is the diagnosis, and its size tells you which repair is proportionate.

Four probes, four layers, one rule: the first probe that fails is the answer, and every hour spent past it is spent on a layer that was working.

The three causes that survive all four probes

Occasionally the chunk exists, has text, passes the filter, answers to its own words and still loses the real query by a wide margin. Three explanations remain, and they are diagnosable.

• Vocabulary mismatch. The document says 'termination for convenience' and users ask about 'cancelling early'. Test it by paraphrasing the query into the document's register: if that version wins, the gap is lexical and belongs to query expansion or a keyword arm, not to the index.
• A chunk covering five subjects. One vector per chunk means the representation is an average, and an average is a weak match for each of its parts — the mechanism set out in one vector per chunk and topic mixing. Split it and re-probe.
• The answer straddles a boundary. The condition is at the end of one chunk and the exception at the start of the next, so neither is a good match for a question about both. Check the two adjacent chunks before concluding anything about the one you started with.

If instead all four probes pass and the document is being retrieved on the real query, the investigation is over and the fault has moved downstream. What happens after correct passages arrive is a different failure family, covered in the right passages and a wrong answer.

Make the probe a tool, not a notebook

This sequence gets run once by hand, then re-derived from memory three months later by someone else, badly. It is a small admin page: paste a document id and a query, get back chunk count, character lengths, filter-on and filter-off results, exact-text rank and the score table. That is an afternoon of work and it is the kind of narrowly scoped system we describe under internal tools and operations software.

The corpus-wide version of this symptom — everything getting slowly worse as documents accumulate rather than one file misbehaving — has a different cause and a different measurement, and it sits alongside this page in retrieval and grounding, within the broader engineering library.

• retrieval
• debugging
• embeddings
• filters