The same customer appears three times under three spellings

The finance report says 412 accounts and the assistant says 389, and both are reading the same warehouse. Nothing is broken. The 2 systems that fed it never agreed that Northgate Logistics Ltd, NORTHGATE LOGISTICS LIMITED and Northgate Logistics (UK) are one company, because they share no identifier, and every layer downstream faithfully preserved that disagreement. A join produced 3 rows. A count produced 3. A retrieval query produced 3 documents that each describe a fraction of one relationship.

It surfaces late and it surfaces as something else. Somebody asks for a customer's total exposure and gets a third of the truth. Somebody exports a mailing list and one office receives it twice. The instinct is to distrust the reporting layer, which is the one layer behaving correctly. The failure is upstream, where 2 records should have become 1 entity and no component was responsible for deciding that.

Size it before anyone writes a matcher

Entity matching is one of the few data problems where the repair is genuinely expensive and the size is genuinely unknown until measured. Teams commission a matching service for a corpus sitting at 0.4 percent, and try to hand-merge one sitting at 11 percent. Half a day of sampling separates those 2 worlds.

1. Pick the entity and the population. One entity type at a time — customers, or suppliers, or sites. Record the exact row count you are reasoning about, per source system, on the day you sample.
2. Normalise into a comparison form, not into the data. Lowercase, strip punctuation and legal suffixes such as Ltd, Limited, GmbH and Inc, collapse whitespace, and keep it as a derived column. You are building a lens, not editing the source.
3. Choose 3 block keys. A block key is a cheap signal 2 records must share to be worth comparing at all: the first 5 characters of the normalised name, the postcode, the email domain, the last 7 digits of a phone number. Each one hides a different kind of duplicate.
4. Generate candidate pairs inside blocks and sample 200 of them, across all 3 keys rather than the productive one, including pairs the blocking barely qualified.
5. Adjudicate every pair by hand with someone who knows the business. Three verdicts only: same entity, different entities, cannot tell without asking the account owner. A large third bucket is itself the finding.
6. Report 4 numbers: pairs generated per block key, the share judged same, the share judged undecidable, and an estimate of records affected rather than pairs affected.

The fourth number is the one quoted upward and the one people compute wrongly. Pairs are not records: one entity fragmented into 4 records generates 6 pairs, so a pair-level figure overstates the human effort. Collapse confirmed pairs into connected groups, then count the records inside groups larger than 1.

Whatever the block key ignores, you will not find

Blocking buys tractability by refusing to compare most pairs, so its blind spot is inherited by every number you produce. Block on the first 5 characters of the name and you will never see the pair where a trading name and a registered name differ from the first character. Block on postcode and you will never see the customer who moved offices between the 2 records being created.

• Name prefix. Productive on formatting variance, blind to abbreviations, initialisms and reversed word order.
• Postcode or city. Catches renamed entities at one address, misses relocations and the registered-agent addresses that hundreds of entities share.
• Email domain. High precision on business customers, useless for consumers on shared providers, and a false-positive generator for groups whose subsidiaries share one domain.
• Phone suffix. The last 7 digits ignore country and area code formatting, where most phone variance lives, and it fails for entities that never supplied a number.
• Sorted-token signature. Sorting the normalised name words alphabetically before taking a prefix catches reversed order, at the cost of more candidate pairs.

Run at least 3 and report per key. If one key produces almost all the confirmed duplicates, that names the dominant cause, which is the next thing you need.

Four ways one customer becomes three records

The third row gets missed because it does not look like a matching problem at all. A merge is an update that changes an identifier, and most change detection watches for content edits rather than identity edits — the same mechanism behind a document updated at source and still served stale. If merges emit no event downstream, the duplicate count keeps growing after a matcher ships: the matcher fixes history and nothing fixes the flow.

Where entity names are read out of documents rather than typed into forms, add a fifth suspect: a damaged extraction manufactures spellings no human ever entered. Screen that slice first, with the legibility pass in PDFs that came through as gibberish. Cleaning names an extractor invented is work you do twice.

Nothing in the stack is wrong. Three records were created, three records were counted, and no component was ever given the job of deciding they were one.

The pairs a human has to reject

A matcher tuned on the assumption that similar names mean one entity will merge things that must stay apart, and a wrong merge is worse than a missed one: hard to detect afterwards, and it moves invoices, cases or entitlements between real parties. Adjudication exists to size this population before a threshold gets chosen.

• Branches and franchises. Two sites of one chain share a name and differ in everything operational, and whether they are 1 customer or 2 is a commercial decision nobody wrote down.
• Parent and subsidiary. Similar names, one shared domain, separate contracts and separate credit exposure. Merging them makes an exposure report wrong in the expensive direction.
• Genuine namesakes. Common personal names and generic trading names collide constantly, and 2 people at one large employer share a name more often than intuition suggests.
• An entity that legitimately split. One record correctly became 2 after a demerger, and the historical similarity is a fact about the past.
• Test and training records. Not duplicates and not real, and they contaminate the measurement in both directions if nobody excludes them first.

Two of these turn on a rule only the business can supply, which is why adjudication needs somebody from operations and not only from engineering. Reading actual records with that person first is the cheapest hour in the project — the habit is set out in reading fifty records before designing anything.

Reading the number you just produced

The last row is the finding people least want and most need. If an adjudicator with the full record in front of them cannot tell whether 2 rows are the same company, no similarity threshold will either; it will make the same guess silently and at volume. The answer is a field captured at entry — a registration number, a verified email, a site identifier — which is a product change, not a data-engineering task.

Whichever band you land in, the merge decisions need somewhere to live and someone to make them. A queue showing a candidate pair, the evidence on both sides and a same-or-different control is small software, and building it before the matcher gives the matcher's output a destination on day one. That is ordinary internal tools and operations software, and worth defending against the claim that a review layer over an existing system is not real product — a position taken apart in what a wrapper is, and when being one is fine.

What the measurement deliberately leaves open

You now have a size, a dominant cause and a rejected-pair population. You do not have a matching design, and choosing one from these numbers alone is the next mistake in the sequence. Scoring functions, thresholds and what a probabilistic match can honestly assert belong to what record linkage can and cannot decide.

Two decisions also stay open. Which system holds the surviving version of a merged entity is governance, answered in the golden record in a messy source set. And when the duplication is in documents rather than rows, the repair differs enough to need two copies of one document and an assistant that believes both. Both sit in data readiness and pipelines, part of the engineering library.

• entity resolution
• data quality
• deduplication
• diagnosis