LLM evaluation in practice: how to know your AI feature actually works

Conventional tests assert that a given input produces a given output. Model-backed features do not work that way: the same input produces a different string each time, and most of those strings are fine. You are not asserting equality — you are measuring a distribution against a standard.

That sounds academic until the first time someone edits a prompt to fix one complaint and silently breaks a category of inputs nobody was watching. The harness below is the minimum that prevents that, and it is smaller than most teams fear.

Step 1 — Collect real cases, not invented ones

The most common mistake is generating the evaluation set with a model. Synthetic cases inherit the model's idea of what your problem looks like, which is precisely the assumption you are trying to test. They produce reassuring scores and catch nothing.

• Start from real traffic, real documents, real tickets — whatever the system will actually see.
• Aim for fifty to two hundred cases. Bigger sets are not better if nobody maintains them.
• Deliberately over-weight the awkward ones: ambiguous inputs, unusual formats, out-of-scope requests, the long tail that generated complaints.
• Include cases where the correct answer is a refusal. A system that never declines is not calibrated.
• Every time something goes wrong in production, the failing case joins the set. This is how the harness stays honest.

An evaluation set built from real failures is worth ten times one built from imagined successes.

Step 2 — Write down what "correct" means

Vague criteria produce vague scores. "Is this a good summary?" is unanswerable consistently, even by people. Decompose it into things that can be judged independently and, where possible, mechanically.

Step 3 — Use a model as judge, carefully

For genuinely subjective criteria, having a model score the output is the practical option. It works well enough to be useful and badly enough to need guardrails.

1. Score one criterion per call. Composite quality scores are noise.
2. Use a coarse scale. Binary or three-point judgements are far more stable than one to ten.
3. Give the judge the rubric and a worked example of each score, not just the label.
4. Validate the judge against human labels on a sample before trusting it. If it disagrees with your domain experts, fix the rubric.
5. Beware self-preference: a model judging its own output rates it generously. Use a different model as judge where you can.

Track judge-versus-human agreement as its own metric. When it drops, the harness is lying to you and the whole edifice is unreliable.

Step 4 — Evaluate retrieval separately from generation

If the system retrieves before it answers, measure the two halves independently. Otherwise a poor score is uninterpretable — you cannot tell whether the model reasoned badly or was handed the wrong material.

Label which passage answers each question, then measure how often it appears in the top-k results. That recall number is a hard ceiling on end-to-end quality, and it is usually where the actual problem is. More on that in RAG vs fine-tuning.

Step 5 — Run it in CI, on every change

An evaluation set that someone runs manually before a big release is a document. One that runs on every pull request is a safety net.

• Run on every change to a prompt, a model version, a retrieval parameter, or a tool definition.
• Fail the build on a regression beyond an agreed threshold, not on any movement — there is inherent variance.
• Report per-category scores, not one aggregate. An average conceals the segment that collapsed.
• Pin the model version explicitly. Silent provider-side updates are a real source of drift.
• Keep it fast enough that people do not route around it. A subset on every commit and the full set nightly is a reasonable compromise.

Step 6 — Close the loop with production signals

Offline evaluation tells you whether a change is safe. It cannot tell you whether the feature is useful. Pair it with signals from real use: how often a suggestion is accepted unedited, how often output is corrected in the review queue, how often users escalate to a human, how often they simply stop using the feature.

When production signals and offline scores disagree, believe production and fix the evaluation set — usually it is missing the cases users actually bring.

The minimum viable harness

If this feels like a lot, start here. It is a couple of days of work and it is most of the value:

1. Fifty real cases in a file, with a labelled expected outcome for each.
2. Deterministic checks for structure, required fields, and refusals.
3. One model-judged criterion for the thing you care about most.
4. A script that runs it and prints per-category pass rates.
5. A rule that every production failure becomes a new case.

• evaluation
• engineering
• testing
• quality