Two copies of the same document, and the assistant believes both

Somebody asks about the notice period and gets 30 days. A colleague asks the same question an hour later and gets 60. Nothing changed in between, no deployment happened, and both answers arrive with a citation that checks out. The corpus contains 2 copies of one agreement: an earlier draft and the executed version, both indexed, both plausible, differing in about a paragraph. Retrieval ranks them almost identically because they are almost identical, and whichever wins the run supplies the number.

This is not the exact-duplicate problem, which is mostly harmless. Two byte-identical files produce 2 chunks saying the same thing; the answer is unchanged and the cost is a wasted slot in the context window. The failure that matters is the near-duplicate, worse precisely because it is nearly the same: the 2 documents agree on everything nobody asked about and disagree on the clause that made someone ask.

Confirm it with one pair before theorising about ranking

The contradiction is easy to reproduce and easy to misattribute. Before touching retrieval settings, prove that 2 sources actually disagree.

1. Capture both answers verbatim, with their citations, and the exact question text that produced each. A remembered question is not the question.
2. Pull the cited chunks and their parent documents. If both citations point at the same document, this is not a duplication problem and belongs elsewhere.
3. Diff the 2 parent documents on normalised text — lowercase, whitespace collapsed, punctuation stripped. Record the similarity and the specific passages that differ.
4. Read the differing passages against the question. A pair that differs only in a footer, a page number or a header block is not contradicting anything; a pair that differs in the notice period is your defect.
5. Check both documents for a date, a version marker or a status field, and note whether the pipeline captured any of it as metadata. Usually it captured none, which is the second finding.

Why the almost-identical copy is the dangerous one

Three mechanisms compound, and none of them produces an error anywhere in the stack.

• Both copies rank. Similarity to the query is driven by the shared text, so 2 near-copies land adjacent in the ranking and small run-to-run variation decides the order. The system is not choosing between them; it is not aware there is a choice.
• Both copies look authoritative. Each is a complete, well-formed document with the same title, the same headings and the same signatories. Nothing in the text says draft, and the reader has no way to tell from a citation which one governs.
• The disagreement is invisible when only one is retrieved. If the answer includes just one passage, the output is confident and unqualified. The contradiction only becomes visible when someone asks twice, or compares notes with a colleague, which is why it is reported weeks after it started.

An exact duplicate wastes a slot in the context window. A near-duplicate spends it on a sentence that was superseded, and cites it.

Where the second copy came from

The second row is a pipeline defect rather than a content one. A source that re-exports its whole corpus under a new folder each run keeps manufacturing copies until change detection is fixed — the mechanisms and their blind spots are in change data capture for a document corpus. It is also the origin most likely to arrive in a burst, which is why it shows up alongside an ingest queue that backs up every Monday: a weekly full export floods the queue and doubles the corpus in the same run.

The fourth row is the one people get wrong in the other direction. A translated or regional variant is not a duplicate to remove; it is a legitimate document needing a language or jurisdiction field so a query can be scoped to it. Merging it away deletes an answer some users need.

The detection pass, and why file hashes miss most of it

A hash of the file bytes catches only byte-identical copies, which is the cheap half of the problem and the half that does not hurt. Two exports of the same document from 2 systems differ in metadata, in line endings and in how the extractor handled a ligature, so their hashes differ while their text is the same.

1. Normalise the extracted text: lowercase, collapse whitespace, strip page furniture such as running headers and page numbers, and drop anything shorter than about 200 characters from the comparison.
2. Compute a similarity signature over word shingles of 5 to 9 tokens. Minhash or simhash both work; the point is a signature you can compare cheaply across the whole corpus rather than pair by pair.
3. Cluster with a threshold band. Above roughly 0.995, treat as an exact copy and keep 1. Between 0.85 and 0.995, flag for review — this is the contradiction band. Below that, leave alone.
4. For every flagged cluster, generate the passage-level diff, because a cluster is not actionable until somebody can see the 40 words that differ.
5. Sort clusters by how often their members are retrieved. A near-duplicate nobody has ever retrieved is a hygiene item; one that answers 200 queries a month is an incident.

Normalisation has to happen before hashing or the pass reports garbage. If part of the corpus was extracted badly, 2 copies of one document will look unrelated because one of them is mojibake — screen for that first, using the check in PDFs that came through as gibberish. Running the same file through ingestion twice should also never create a second record in the first place, which is the guarantee described in idempotent ingestion.

Which copy governs is a decision, not a score

This is the part that cannot be automated, and the part teams try hardest to automate. No similarity measure, recency heuristic or file-path convention can determine which copy of an agreement is the one in force. Later is not automatically authoritative: an amendment can be superseded, a re-export can carry an old modification date, and a scanned executed copy is frequently older on disk than the unsigned draft that produced it.

So the detection pass produces a decision queue, not a deletion list. The rule its owner writes down has to be specific enough that a second person applies it identically: which system of record wins, what marks a document as executed, what happens when a cluster holds a version in another language. Standing that role up is the subject of giving the knowledge base an owner. It is fair to raise early when choosing an AI development partner — a partner who quietly picks the most recent file has made your policy for you.

What happens to the copy that loses

Deleting it is the obvious move and usually the wrong one. The losing copy is often evidence — what a clause said before amendment is exactly the question a dispute turns on — and a deletion that removes it from the source system as well as the index is not something you can undo when the ruling turns out to be wrong. Run the reversibility check in checking a decision is reversible before making it before any bulk action touches a source system.

• Mark, do not remove. A status field of superseded, with a pointer to the governing copy, keeps the document readable and stops it competing.
• Filter at query time by default. Excluding superseded documents from the candidate pool is a metadata filter, and it is reversible in a way that deletion is not.
• Show the relationship in the answer. When a retrieved passage sits in a cluster, saying so — and naming the governing version — is more useful than silently suppressing one side.
• Keep the cluster, not just the winner. The record of which documents were judged copies of each other is what lets somebody re-examine the ruling later without redoing the analysis.

The suppression rule is also the piece most likely to be built as ordinary application logic rather than model work: a status field, a filter and a small review interface. That is squarely AI agents and automation territory in the unglamorous sense — most of the value is in the workflow around the model, not in the model.

What clearing the duplicates will not fix

Two failures survive a perfect deduplication pass and get misattributed to it. The first is a single document that contradicts itself, where an amendment was appended to the original rather than replacing it; there is one file, one hash, one cluster of size 1, and the contradiction is internal. The second is entity-level duplication in structured data — 3 spellings of one customer, none of them documents — which is a different diagnosis with a different sizing method, set out in the same customer under three spellings.

It also does not fix retrieval that prefers an old version over a new one when both are legitimately present, which is a ranking problem rather than a corpus problem. That belongs to the retrieval cluster. This page and its neighbours sit in data readiness and pipelines, part of the engineering library.

• near duplicates
• knowledge base
• document authority
• corpus hygiene