One intent label is absorbing most of your tickets

Before anyone retrains anything, pull two distributions for the same 30 days of tickets: the label the router predicted at arrival, and the label those tickets carry now, after agents corrected them. A distribution that is skewed but agrees with the corrected version means your queue is skewed and the model is right. A distribution that disagrees tells you where, and the shape of the disagreement names the cause.

This ordering matters because the instinctive response — more training data, a bigger model, another round of labelling — fixes exactly 1 of the 4 causes, and is the most expensive of the available moves.

The confirming check: two histograms, one query

1. Count predicted labels for every ticket created in the last 30 days, as recorded at creation. If you are not storing the prediction separately from the current value, that is the first thing to fix — you cannot diagnose this without it.
2. Count the current label on those same tickets, after correction and closure. Same population, same window, different column.
3. For each label, compute three numbers: predicted share, corrected share, and correction rate — the proportion of tickets originally given that label which now carry a different one.
4. Split both by week and by arrival channel. A collapse that begins on a specific date is a configuration change; one that is flat across the window is a definition problem.
5. Check the deployment and settings log for that date before forming any theory. Threshold edits, taxonomy edits and prompt edits are rarely announced and are the single most common trigger.

The four causes, ranked by how often they are the answer

In order of frequency, not severity. Work down, and stop when the evidence matches — they produce different histograms, so you rarely need to guess between two.

1. Overlapping definitions. Two labels are both defensible for the same ticket, annotators split, and the model resolves the ambiguity toward the larger prior. Symptom: high correction rate, stable distribution, and 2 reviewers disagreeing on a sample of 30 tickets. Fix: write one boundary sentence per label pair, then re-annotate.
2. The label meant something else when the training data was cut. Definitions drift silently — a billing label that once covered failed payments and now excludes them. Symptom: accuracy that was fine at launch and decays over months with no incident. Fix: date your label definitions and re-cut the training sample from after the last change.
3. A confidence floor routing everything below it into one bucket. The bucket's size is then a function of the threshold, not of the tickets. Symptom: a step change on a date, concentrated entirely in the fallback label. Fix: read the setting, do not retrain.
4. Genuine skew. A pricing change, a failed integration or an incident produced real volume of one kind. Symptom: predicted and corrected histograms agree, and the affected tickets are legitimately about that thing. Fix: nothing. Accept it, and consider whether the taxonomy still earns its granularity.

There is a fifth possibility that is not a classification fault at all: the input is contaminated. If one customer's thread fragments into 6 tickets, all 6 get the same label and the histogram tilts — the transport-level cause is in customer replies opening brand-new tickets. A more dramatic version is two automated systems answering each other and manufacturing near-identical tickets by the hundred, diagnosed in a mail loop between your system and theirs. Both look like a collapsed classifier and neither is one.

The threshold is the cause that most looks like a model problem

Any sensible router abstains: below some confidence it declines to guess and writes an unclear label instead. That is correct behaviour, and it means the size of the unclear bucket is set by a number in a config file. Move the floor from 0.55 to 0.75 and the bucket can double overnight without a single weight changing.

• Give the abstention bucket a target size and tune the threshold to it. Somewhere under 10 per cent of arrivals is a reasonable starting point for a mature taxonomy; higher than that and humans are doing the routing.
• Route abstentions to a staffed triage view, not to a general queue. An unclear label that nobody owns is worse than a confident wrong one, because at least the wrong one is visible to somebody.
• Sample the abstentions weekly. If a recognisable topic keeps appearing there, the taxonomy has a hole and no threshold will fix it.
• Alert on the derivative, not the level. A bucket that is 12 per cent every week is a design decision; one that went from 4 to 19 per cent on a Tuesday is an incident.

The size of your unclear bucket is a number somebody typed into a settings page. Check that before you commission a retraining run.

The decision tree

1. Do the histograms disagree? If not, accept the skew and go no further. The model is reporting your queue accurately.
2. Is the excess concentrated in the abstention label? Re-threshold. Read the change log first, then tune to a target bucket size.
3. Is the correction rate high while the distribution stays stable? Relabel. The definitions overlap, so write boundary sentences and re-annotate before touching the model.
4. Did the collapse start on a date? Find the change. A taxonomy edit that merged or renamed a label invalidates every prediction made under the old meaning.
5. Is one channel responsible? Fix the input. Duplicates and loops are cheaper to stop at the transport layer than to unpick downstream.
6. Only if none of the above holds: re-scope. Cut a fresh training sample from after the last definition change, and evaluate on macro recall rather than accuracy.

When the answer is accept

A support queue is not uniformly distributed and never will be. If the largest label is genuinely the largest thing customers write about, a taxonomy that reflects that is doing its job, and splitting the label to make a chart look balanced makes annotation harder for no operational gain. Split it only when the child would route somewhere different, to someone different, or with a different target.

That still leaves a real need: understanding what is inside the big label. Answer that with themes rather than intents — the reporting path from a single sentence in a ticket to something a product team can act on is from a customer sentence to a roadmap candidate. Intents route; themes explain; using one for the other is what produced the oversized label set in the first place.

One more thing this does not fix. Correcting a label does not settle who owns the ticket, and a well-labelled ticket can still ping-pong between teams if 2 rules both claim it — that is tickets bouncing between two groups, a routing conflict with its own diagnosis.

Instrumenting a classifier so this question is answerable in 5 minutes rather than a week is a small build: store the prediction, store the confidence, store the corrected value, and chart the three. That instrumentation is part of every triage system we ship in ticket and inbox triage, inside our SaaS and customer support practice, and it is ordinary MVP and product build work rather than data science.

• intent classification
• triage
• model evaluation
• taxonomy