Draft acceptance rate: three outcomes it has to separate

Draft acceptance rate is the share of generated drafts that an agent actually used, and it is worth reporting only when it splits into 3: sent with no change, sent after the agent edited it, and discarded. Those are 3 different verdicts on the system. One says the draft was right; one says it was close and names what was wrong; one says it was useless or slower than typing. Averaged into a single percentage they cancel each other out.

The middle outcome is the one worth building instrumentation for. A discarded draft tells you something failed; an edited draft tells you exactly which sentence failed, in the agent's own words, at no labelling cost. That is why acceptance is a measurement question before it is a model question. What a draft has to carry for any of it to be recordable is set out in what a drafted reply must contain before it is sent.

Three verdicts, and the event that proves each one

Note the last 2 rows. They are not outcomes of a draft; they are outcomes of the system deciding whether to draft at all. Folding them in is the single most common way an acceptance figure becomes meaningless, and it moves the number in whichever direction the person reporting it prefers.

Five ways the denominator quietly flatters you

• Unopened drafts excluded. The population becomes drafts agents already chose to engage with, which is circular — and it is why very high figures should be read as a bug report about the query.
• Internal notes counted with public replies. A note has a different audience and a different failure cost — see the internal note the customer never reads. Mixing them makes the rate move when the mix moves.
• Regenerations counted as separate drafts. One agent pressing regenerate 3 times before sending reads as 33 per cent or 100 per cent depending purely on how you group.
• Merged and closed tickets left in. A draft on a ticket that was merged away was never sendable by anyone, and belongs outside the population rather than in the discard bucket.
• One pilot cohort standing in for the team. Volunteers accept more; a fall at general rollout is composition, not regression — as the staging in introducing drafts to a team that did not ask for them expects.

Four cases where the outcome is genuinely arguable

1. The agent copies the draft into a fresh compose box. Textually that is acceptance; in the click stream it is a discard followed by an unrelated reply. Resolve it on text similarity between sent body and offered draft, with 1 threshold for the whole team.
2. The agent pastes half of it. Count it as an edit with a large distance, not a discard. A correct explanation with a wrong opening is a voice problem, not a knowledge problem.
3. The agent was simply faster. On short intents a good agent types the reply before the draft renders. Log time-to-draft: a discard at 400 milliseconds and one at 9 seconds are different events.
4. The draft was accepted and the ticket reopened. Acceptance is a 5-second judgement under queue pressure, not a correctness verdict. Reopen rate has to be reported against the same drafts.

Edit distance, and why its shape matters more than its mean

Once the 3 outcomes are separated, the headline rate stops being interesting and the edited bucket starts. The measurement is the distance between the generated text and the sent text, and the useful form is a distribution, not an average.

1. Normalise first. Strip the helpdesk's appended signature, the templated greeting and any merge fields rendered after generation. Skip this and every draft looks edited, because the platform edited it.
2. Diff at token level, not character level. Levenshtein over characters reports a rewritten sentence and a corrected spelling as comparable events; token-level diff separates them.
3. Bucket rather than score. Cosmetic, substantive and rewrite is enough — 3 buckets with a written boundary beats a continuous metric nobody can interpret at a standup.
4. Plot the distribution weekly per intent. A single mean hides the only pattern that matters: 2 peaks means 2 different problems, typically a voice problem on short replies and a knowledge problem on long ones.
5. Cluster the identical edits. When the same deletion appears across dozens of drafts it is a corpus artefact, not agent preference — the fix is in agents deleting the same sentence from every draft, and no amount of prompt tuning replaces it.

An edited draft is a labelled example somebody produced for free, under real conditions, with the customer in front of them. There is no evaluation set you can buy that is worth more.

Reading a suspiciously good number

A rate in the high nineties is almost never a strong model. In order of likelihood it is a denominator excluding unopened drafts, a population narrowed to 1 easy intent, a comparison that normalises away real edits, or a team that has learned drafts are safer to send than to argue with.

All 4 are answerable from event data, if the events exist. Acceptance is reconstructed from the draft record, the compose events and the outbound message, on the same log as everything else in drafted replies and agent assist — the case for treating that log as the system of record is the audit trail is the real record, not the ticket. Wiring those events is a small MVP and product build inside our SaaS and customer support work, and it separates a number you can act on from one you can only quote.

• measurement
• agent assist
• acceptance rate
• edit distance