Every escalation creates a second enquirer record in the CRM

The duplicates are created at the moment of the write. An assistant that opens a CRM contact on the first inbound message has a display name and a channel handle and nothing else, so it cannot match, so it creates. Every subsequent escalation from the same person repeats the act, and no matching rule added later can recover a key that did not exist when the row was made.

It shows up as 3 separate complaints that nobody connects: contact counts rising faster than genuine enquiry volume, families receiving the same nurture mail twice in a week, and an advisor opening a record that holds a third of a conversation they know went on longer. All 3 are the same defect seen from different desks.

Pull the duplicate clusters with the write that made them

The point of this check is not to count duplicates. It is to see, for each cluster, which key was available at the instant the second row was written — because that is what names the fix.

1. Export every contact created in the last 90 days with its created-at timestamp, the integration or user that created it, the channel, and the raw name, email and phone as captured.
2. Group candidates 3 ways and keep the groups separate: on normalised email, on the last 8 digits of the phone, and on a normalised name plus programme of interest. Different keys expose different causes, and merging the groups hides that.
3. For each cluster, sort by created-at and record what identifiers the earliest row held versus the later ones. A first row with no email and a second row with one is a deferred-identity problem, not a matching problem.
4. Tag each cluster by creating source: assistant, web form, agent portal, import, staff entry. If 1 source dominates, the whole investigation collapses to that integration.
5. Count records inside clusters larger than 1, not clusters. A person fragmented into 4 rows is 1 duplication problem and 4 pieces of duplicate mail, and the second number is what operations feels.
6. Sample 20 clusters and read them. About a fifth will turn out to be genuinely different people, and knowing your own false-positive rate before writing a matching rule is what stops the rule merging 2 siblings.

Five moments where the key was missing

No component behaved badly. A record was created before anybody knew who it was for, and everything downstream faithfully preserved that ignorance.

One person, three id spaces, no thread between them

The most expensive duplicates are not 2 rows a week apart. They are the same human appearing as an enquirer in the spring, an applicant in the autumn and a student the following year, in 3 systems that each issue their own identifier and none of which was told the 3 are one person. The enquiry record is orphaned first, which is why nobody can answer whether the open day converted anything.

• Enquiry identity is self-asserted. Whatever the person typed, unverified, possibly a nickname, frequently a parent acting on their behalf.
• Applicant identity is issued by the application route. A direct portal account, an agent submission, or a national application service that issues its own applicant identifier — and a person who applies twice may hold more than 1.
• Student identity is issued at enrolment by the student information system, usually a completely separate number space, often with its own email address created at the same moment.
• The join between them is a decision nobody is asked to make. Pick which id is durable, issue it at the earliest point identity is confirmed, and carry it forward as an attribute of the later records rather than replacing them.

Defer, match or stage: choosing where the write goes

There are 3 designs, and the choice is a business decision about what you lose rather than an engineering preference.

Staging is the right default for an assistant. It keeps the transcript, it never loses an escalation during a peak, and it stops the contact table filling with rows that represent nobody. It also gives the merge queue somewhere to send rejects instead of leaving them in the live table — the small piece of software that makes any of this operable, and the kind of thing we scope as an MVP and product build rather than a platform replacement.

Normalising a name is a policy decision, not a string operation

Every institution recruiting internationally has to decide, in writing, how names are held — and until it does, each entry point invents its own answer and the duplicates follow automatically.

• Word order. Family name first is correct in several of the regions most institutions recruit from, and a form labelled first name and last name silently reverses it for a large share of applicants.
• Transliteration. A name written in 1 script on a passport and another on an email signature is 2 strings for 1 person, and the mapping between them is not reversible. Where the same content has to survive 2 languages, the pipeline question is the same one as the translated answer that contradicts the English policy page.
• Diacritics and case. Strip them for the comparison key, never in the stored value: a student's name displayed without its accents is a small, repeated insult that also breaks their document checks.
• Honorifics, patronymics and generational suffixes. Store them as separate attributes rather than letting them contaminate the name field and split the person.
• Compare on a derived key, store the original. Normalisation belongs in a comparison column so the record still shows what the applicant wrote — the same principle as a system quietly rewriting the conventions its users depend on, which is the failure in the tutor using notation the course does not use.

What merging the duplicates will not undo

A merge repairs the table. It does not repair what the table already caused, and 3 things survive it. Mail already sent stays sent, and a family that received 2 identical sequences remembers that rather than the merge. Communication preferences and consent recorded against the losing row have to be carried across explicitly, or an unsubscribe silently becomes a resubscribe. And any reporting already published on inflated counts stays published, so the corrected figure needs an explanation before somebody reads it as a collapse in demand.

Volume also makes the underlying fault worse rather than more visible: results week and add-drop week produce the most escalations, the least identity information, and the highest chance that a family is contacting you from 3 devices and 2 addresses. Duplicate rate belongs on the pre-peak list in readiness checks before results day and add-drop week, measured before the surge rather than reconstructed after it.

None of this touches the read side. Whether the assistant's answers are correct is a corpus problem with its own method, in preparing a catalogue and handbook set for retrieval. Both sit under admissions, enrolment and student-services assistants, within our education and edtech work.

• admissions CRM
• deduplication
• integration
• diagnosis