The draft promised a refund window that does not exist

A draft asserted a term nobody at your company ever wrote — a refund window, a credit, a cancellation right. Before touching the prompt, run one search: look for that exact term across every source the drafting system can read, including retired documents and every localised version. What comes back tells you which of 3 defects you have, and they need 3 different fixes.

This matters more than the average drafting bug because of what a support reply is. When an agent sends it, the company has said it. Whether a model composed the sentence is a fact about your tooling, not a defence, and it is why a human-in-the-loop review layer exists at all.

One search, 3 possible answers

1. Search every readable source for the specific claim — help centre, internal policy store, macro library, past sent replies, terms pages, and anything in the retrieval index that is no longer published anywhere a customer can see.
2. Search for the concept as well as the wording. A model that wrote "within 30 days" may have grounded itself in a document saying "one month", which is a match for this purpose.
3. Check the localised and market-specific copies separately. A term that exists only for one region, retrieved for a customer in another, is a corpus scoping problem rather than an invention.
4. Check the account state the model was given. A correct policy applied to a stale entitlement produces a false sentence built entirely from true inputs.
5. Record which of the 3 outcomes you got — absent, present but wrong, or present and contradicted — because the remedy branches here and nowhere else.

That last row is missed constantly, because everything reads correctly in isolation. The policy is accurate, the sentence is grammatical, and the plan name in the prompt was simply hours out of date — the sync failure argued in polling against webhooks for keeping tickets in sync. Check what the model was told before you conclude it made anything up.

Why absence produces confident text rather than silence

A generator asked to answer a question it has no source for will still answer, because nothing in the objective rewards refusal. It fills the gap with the most probable version of the thing — and for a product in your category, the most probable refund window is whatever your competitors offer. The output is fluent, plausible and specific, which is precisely the combination a reviewer under time pressure accepts.

A model has no way to distinguish a policy you have from a policy a company like yours would probably have. Only your corpus can make that distinction, and only if the system is required to consult it.

So the durable fix is structural: make the ungrounded sentence impossible to emit, rather than instructing the model not to want to. An instruction is a preference the model weighs against everything else in its context. A check that runs after generation is not.

The words that may not appear without a source attached

You do not need a citation on every sentence — that produces a wall of markers agents stop seeing, and it slows drafting for no benefit. You need one on the sentences that can bind you. That set is small enough to enumerate, and enumerable is what makes it enforceable.

• Any sentence containing a number or a duration. Days, months, percentages, counts, limits and thresholds — the shape of nearly every commitment a support reply can make by accident.
• The commercial verbs: refund, credit, waive, cancel, extend, upgrade, downgrade, guarantee, compensate. Each names an action with a cost attached, and none of them should ever be improvised.
• Eligibility and entitlement language — "your plan includes", "you qualify for", "this is covered" — which asserts a fact about the account rather than about the product.
• Future-tense operational claims. "This will be fixed in the next release" is a commitment on a team that did not agree to it, and it is the one support most often makes on engineering's behalf.
• Anything naming a legal or regulatory position. Support drafts are not the place to characterise a right or an obligation, and a model asked to will produce something confident and unverifiable.

This is also the honest boundary for macros. Wording that is legally or procedurally fixed should stay fixed text a human wrote and approved, not something regenerated each time — the line drawn in keeping the macro library or letting it become the training input. Everywhere else, a generated sentence with a citation beats a macro nobody has read since it was written.

Check after generation, and reject the sentence rather than the draft

Put a deterministic pass between the generator and the compose box. It does not need to be clever, and it should not be a second model asked politely to review the first.

1. Split the draft into sentences and flag every one that matches the commitment vocabulary. Pattern matching is enough here, and its predictability is a feature — an agent can learn what will be flagged.
2. For each flagged sentence, require an attached source reference that the retrieval step actually returned. No reference, no send, regardless of how confident the generator reports itself to be.
3. Verify entailment, not overlap. The cited document must support the specific number or term in the sentence; a document about refunds generally is not a source for a 30-day window.
4. Replace the failing sentence rather than the draft. Strike it, mark the gap visibly, and let the agent write that one line — an agent who loses only the sentence keeps the other 90% of the draft, and does not learn to disable the check.
5. Log every rejection with the sentence and the reason. Those logs are the shortest route to the missing policy documents, because they are a list of questions your corpus cannot answer.

The same architecture catches a related class of defect at the same point, which is a good argument for building the pass once and reusing it: the pre-send validator that refuses any outbound text containing unresolved delimiters, described in an unrendered placeholder shipping in a live reply, belongs in exactly this position in the pipeline.

When the corpus is wrong, do not patch it in the prompt

If the search found the term and it was stale, the model behaved correctly. Adding an instruction that overrides the document is the worst available fix: you now have 2 sources of policy, one of which is a prompt, and the next person to change the policy will change the document and wonder why nothing moved.

Fix it at the document, then confirm the index actually picked the change up — a corrected policy that is still serving from a stale index is indistinguishable from no fix at all. Where 2 documents contradict each other, resolve the contradiction rather than tuning retrieval to prefer one: a system that relies on ranking to pick the right policy will eventually rank the other one first.

What the guardrail will not catch

It catches assertions, not implications. "I have escalated this to the engineering team" contains no number and no commercial verb, and it still creates an expectation you may not meet. Tone can commit too, and a sufficiently warm draft implies more than its sentences say. Neither of those is reachable by pattern matching, and both are reasons the named human sender stays in the loop rather than being optimised away.

It also will not tell you which policies you are missing until customers ask. The rejection log is the closest thing to that list, which is why it is worth keeping and reading monthly. Building the check, the source-attachment rule and the logging is MVP and product build work in our SaaS and customer support practice, and it sits alongside the rest of drafted replies and agent assist.

• drafted replies
• grounding
• policy
• guardrails