Agents are deleting the same sentence out of every draft

If every agent removes the same sentence from every draft, the model is not choosing to write it. Something is feeding it that sentence, and it is one of 4 sources: retired macro text still sitting in your sent-reply history, a signature or escalation line your template layer adds after generation, a policy sentence that changed while the corpus did not, or a knowledge article whose preamble the retriever keeps returning. Find the source, delete it there, and the phrase stops without anyone writing a rule.

The instinct is to add a line to the prompt saying never write this. That works, narrowly, for one phrase, and it fails as a strategy: the source is still emitting, the next retired phrase arrives without warning, and your instruction set grows a section of negations nobody can maintain. Worse, a negative instruction suppresses the wording without touching the belief underneath it, so the model finds a paraphrase and agents start deleting that instead.

Rank the deletions before you touch the prompt

The measurement is cheap and you already have the data, provided the generated text was stored alongside the sent text. If it was not, that is the first fix, because without both halves nothing here is possible.

1. Pull at least 200 draft-and-sent pairs from a recent window, spanning every intent your router assigns. A single queue's worth will find that queue's boilerplate and miss the rest.
2. Diff each pair at sentence level rather than character level. You want the units a human removed, not the token churn produced by rewording around them.
3. Keep only the deletions — text present in the generated version and absent from the sent one — and normalise them: lowercase, collapse whitespace, strip the customer's name and any merge output.
4. Count n-grams of 5 to 8 words across that deletion set, and rank by the number of distinct tickets each appears in rather than by raw count. One agent working 30 tickets with the same macro should not outrank 30 agents each hitting it once.
5. Compute the top phrase's share of all deletions. That share is the number the rest of this page turns on.

Reading the shape: who deletes it, and where it sits in the draft

The frequency ranking tells you there is a source. Two more cuts of the same data tell you which source, before you open any of them.

Cause 1: a macro you archived that is still in 3 years of history

The commonest cause, and the one teams rule out last because the macro was deleted from the macro library and everyone remembers deleting it. Archiving a macro removes the button. It does not remove the tens of thousands of replies the macro produced, and those replies are what your drafting system learned voice and content from.

The tell is a date. Plot the phrase's frequency across sent replies by month and you will see a plateau that ends abruptly, and the end will match the day the macro was retired. Everything before that date is in your corpus with equal weight, so a phrase that was standard for 3 years outvotes 6 months of replies that avoid it.

The fix is corpus surgery, and it needs a join key: you can only exclude a macro's descendants if the export carries the macro or template id against each reply. Most do, and most teams drop it during ingestion because nobody thought it mattered — one more reason to decide these mappings deliberately, as mapping helpdesk fields onto a schema you control argues. Failing that, exclude by date window and phrase match, and accept that you are removing some good replies with the bad.

Cause 2: the model writes the sign-off and the template adds it again

If the deleted line is always in the same position — the last sentence, or the greeting — the model is not the only thing writing it. Your helpdesk is appending a signature, an escalation notice or a satisfaction-survey line at send time, and the model, having learned from replies that contained those lines inline, produces its own copy above it.

Agents then delete one of the 2, usually the model's, which is why the deletion looks like a style objection when it is actually a duplication. This is the same boundary problem as a merge field reaching the customer unrendered: the model's text and the template engine's text are being composed without either side knowing what the other contributes.

• Strip template-owned content from the training and exemplar corpus, so the model never learns that a signature is part of a reply body.
• Tell the generator what will be appended, as a structural fact rather than a prohibition — it needs to know the sign-off exists, not to be told 12 times not to write one.
• Render the full outbound message in the compose box, template additions included, so the agent sees the duplication rather than discovering it after send.

Cause 3: the sentence was true, and then the policy changed

Here the deletion is not cosmetic. The model is asserting something that used to be accurate — a window, a fee, an eligibility rule — and agents are removing it because it is now wrong. This is the most urgent of the 4, because the deletion rate is also the miss rate: every draft an agent rushed through carries the old claim out to a customer, and a drafted commitment that is sent becomes a commitment the company made. That is the failure argued in the draft that promised a refund window that does not exist.

A phrase agents delete 90% of the time is not a style problem. It is a defect shipping to customers 10% of the time.

The corpus fix here is different from cause 1. You do not want to remove the historical replies — they are correct for their period — you want the current policy to win. That means the authoritative policy source has to be retrievable, versioned with effective dates, and weighted above reply history for anything factual, with reply history contributing voice and not content.

Cause 4: the article's throat-clearing arrives with the answer

Knowledge articles open with framing that exists for a reader browsing a help centre: a restatement of the question, a note about which plans this applies to, a link to a related article. When a chunk of that article is retrieved, the preamble comes with it, and the model, having been told to ground its answer in the retrieved text, reproduces the framing along with the substance.

The signature is that the deleted phrase appears in your knowledge base but almost never in historical replies, and only shows up in drafts for tickets that hit one particular article. Fix it at chunking time: chunk on semantic units rather than fixed character counts, and strip navigational and preamble text before indexing. Note that this is a retrieval-quality problem and not a citation problem — a draft that cites an article which does not support its answer is a separate defect, handled in the citation that points at an article saying otherwise.

Remove it, exclude it, or write the rule — in that order

The 3 available fixes are not equivalent, and preferring the wrong one is how a drafting system accumulates instructions nobody dares delete.

1. Remove it from the corpus. The only fix that also removes every paraphrase and every future variant, because the model stops believing the phrase belongs there. Requires re-indexing or re-training, so it is the slowest and the most permanent.
2. Exclude it at retrieval time. Filter the offending documents or chunks out of the candidate set. Fast, reversible, and appropriate when the source is legitimate text that simply must not be quoted — an internal article, a superseded policy version.
3. Add a voice-specification rule. Correct when the phrase is genuinely a style choice rather than an artefact, and when it will still be a style choice next quarter. Treat it as the fallback: every rule is a permanent tax on every generation, and nobody ever audits the list.

The systems that hold up over time are the ones that run on data the operation already emits, which is the argument AI in logistics operations makes about telemetry. Sent replies are that data for a support team — and the same property that makes them valuable is why they carry every sentence the team ever retired.

What removing the sentence will not do

It will not raise acceptance on its own. One phrase removed from a draft that is still too long, too formal or wrong about the account is a draft an agent still rewrites, and length has its own diagnosis and its own fix. Nor does it make the deletion signal go away, which is the point: rerun the n-gram ranking monthly, because the next retired phrase is already accumulating and this measurement is the only thing that will surface it before a customer does.

Keeping the generated text next to the sent text is what makes all of this possible, and it is worth saying plainly that the diff is the most valuable evaluation data a support team owns — nobody had to label it. Building that capture, the ranking job and the corpus controls is AI agent and automation work in our SaaS and customer support practice, and it belongs with the rest of drafted replies and agent assist.

• drafted replies
• corpus hygiene
• retrieval
• evaluation