Chunk overlap: what it protects against and what it duplicates

Chunk overlap is the amount of text repeated at the start of each chunk from the end of the one before it, stated in characters, tokens or as a percentage of chunk size. It exists for one reason: a fixed-length boundary knows nothing about meaning, so eventually it lands between a rule and its exception.

The mechanism is a sliding window. Chunk size minus overlap gives the stride — how far the window advances. With 800-token chunks and 200 tokens of overlap the stride is 600, so every 200 tokens appears in 2 chunks. Any span shorter than the overlap sits intact inside at least one chunk, and that is the entire guarantee on offer.

What the insurance actually covers

A policy states a retention period; the exception for one document class begins in the next sentence; the boundary falls between them. Now no chunk answers "how long do we keep these", and retrieval returns something plainly on topic that produces a confidently wrong answer.

Overlap turns that silent miss into a clean hit, provided the idea is shorter than the overlap. It does nothing for a definition and a qualifier 3 pages apart, which is a structure problem no window size solves.

The two costs, one arithmetic and one behavioural

Column 2 drives embedding cost, index size and re-index time, and it is predictable enough to plan around. The second cost surprises people. Prompt slots are zero-sum, so 3 chunks sharing 70 percent of their text win or lose together — and when they win they take 3 slots, contribute one idea, and displace the passage holding the other half of the answer.

So the symptom of too much overlap does not look like duplication. It looks like an answer that is confidently partial. Measuring it means measuring recall@k with one extra column: how many of the k slots held distinct source spans rather than distinct chunk ids.

How you suppress the copies matters. Admitting only the first chunk per document silently removes real second answers from long files; demoting later chunks from the same source still lets a strong one through. That distinction is the subject of metadata filters and the difference between filtering and ranking, and reaching for the wrong one turns a duplication problem into an empty-results problem.

When overlap is the wrong instrument

Overlap is a workaround for boundaries chosen by a character counter. Where a document declares its own boundaries, the workaround is unnecessary and actively harmful.

• Contracts and policies, where the clause is the unit. Split on the clause tree and every chunk is a complete thought with a citable identity; overlap makes citations ambiguous.
• Question-and-answer content, product records and reference entries. Each is short and self-contained, so overlap simply indexes each answer twice.
• Tabular data, where the row is the unit. A row needs its header repeated, which is a deliberate prefix, not a side effect of a sliding window.
• Transcripts with speaker turns, where a mid-turn split loses the attribution that makes the line mean anything.

There is a structural alternative. Index small, precise chunks with no overlap, and at read time serve the matched chunk with its neighbours from the source document. It costs one extra read and a citation that names a span.

Choosing a number on your corpus rather than copying one

1. Take 30 real questions the system answers badly and read the retrieved passages. Count how many failures are boundary splits rather than the wrong document entirely.
2. If that count is under about 3 in 30, overlap is not your problem and tuning it will burn a week.
3. If it is higher, sweep 0, 10 and 25 percent, re-index and rerun the same 30 questions — recording how often the answer was present and how many slots held distinct source spans.
4. Take the smallest overlap that removes the splits. Everything beyond it is index growth bought with no return.
5. Write the residual down. Some boundary miss survives any setting, and naming that rate turns it into a budgeted property — the argument in an error budget for a feature that is sometimes wrong.

Overlap adds no information to the index. It adds copies, and buys the chance that one copy holds a whole idea.

Treating chunking as something measured rather than configured once is most of the difference between a retrieval system that improves and one that gets nudged — a habit running through our AI agents and automation work and what an AI-native product studio does differently. Neighbouring terms sit in retrieval and grounding, in the engineering library.

• chunking
• indexing
• retrieval
• embeddings