The pause after the guest stops talking is where the order is lost

A caller says 'a large margherita' and then hears nothing. At 800 milliseconds the silence is unremarkable. At 2 seconds they say 'hello?'. At 4 they assume the line is dead and hang up, and the order that was 20 seconds from being placed is gone. Nobody logs that as a latency failure; it is logged as an abandoned call.

The useful move is to stop treating response time as one number. What the caller experiences is the gap between the end of their speech and the first syllable of the agent's, and that gap is the sum of 5 stages with wildly different sizes. Optimising the wrong one is the normal outcome, because the stage teams reach for first — the model — is rarely the largest.

Trace one call with the stages separated

1. Log 6 timestamps per turn: last audio frame received, end-of-speech declared, final transcript ready, catalog response returned, first model token, first synthesised audio frame sent. Everything below is a subtraction between 2 of these.
2. Number the turns. Turn 1 is the greeting, turn 2 the first item, and so on, and store the index on every event so you can group by it later.
3. Bucket abandonment by turn index rather than by call. A call that ends at turn 2 and a call that ends at turn 7 are 2 different bugs.
4. Mark which turns issued a catalog lookup and which did not. This single flag usually settles the diagnosis on its own.
5. Repeat the trace at peak. A median measured at 15:00 tells you nothing about 19:30, and the stages do not degrade proportionally — some are flat under load and others are not.

The budget, allocated stage by stage

This is a budget to design against, not a measurement of your system. Allocate it, then hold each stage to its line. A turn that changes the order can afford more than a turn that answers a question, but not much more.

The right-hand column is the diagnostic. A delay present on every turn is upstream of meaning — endpointing, synthesis, network. A delay present only on turns that name a dish is the catalog. That distinction takes one afternoon to establish and it removes most of the guesswork.

Why the menu is the stage that breaks

A restaurant catalog is small — a few hundred items, their modifier groups and their prices, which is comfortably under a megabyte of structured data. It has no business being fetched over the network once per turn, and yet that is the default shape of most integrations, because the POS is the source of truth and calling it feels safer than caching it.

• Hold the menu in process memory, keyed by a version stamp. Refresh on publish and on 86 events rather than on a timer, so the copy is current without a request per turn.
• Never send the whole menu to the model on every turn. Retrieve 10 to 20 candidate items and send those. A full re-read grows the input on every turn of a long call, and the growth is invisible until a 6-item order takes twice as long as a 2-item one.
• Separate the read path from the write path. Availability and pricing can be answered from a cached snapshot; only the final order injection needs to reach the POS synchronously.
• Measure the lookup at the 95th percentile, not the median. One slow query in 20 is one abandoned call in 20, and the median hides it completely.
• Watch what 86'ing does. A live 86 event has to invalidate the snapshot quickly, and a build that solves latency by caching for an hour has traded a slow agent for one that sells food the kitchen does not have.

A delay on every turn is upstream of meaning. A delay only on the turns that name a dish is the menu.

End-of-speech detection is a product decision

Before any of the work starts, something has to decide the caller has finished. Defaults inherited from dictation tooling wait for a long trailing silence — often close to a second — because in dictation a premature cut is worse than a slow one. On an ordering call the trade runs the other way: that wait is added to every single turn, and it is the cheapest place to find 400 milliseconds.

The cost of tightening it is interruption. A caller reading an order off a group message pauses mid-sentence, and a short threshold will cut them off. The workable compromise is to vary it: a short threshold on short, confirmable answers such as a size or a yes, and a longer one when the caller is mid-list. Barge-in matters just as much — if the caller can talk over the agent and be heard, an over-eager cut costs a fraction of a second instead of a whole turn.

Cold starts, forwarding and the top of the rush

Two effects concentrate at exactly the wrong time. The first is the cold start: after a quiet mid-afternoon, the first calls of the evening pay for container start-up, connection pools and caches that emptied. Keeping instances warm across a trading day, or pre-warming 15 minutes before service, removes a class of failure that only ever hits the first callers of the rush.

The second is the call path. A number that forwards through 1 or 2 legacy hops before reaching the platform adds one-way delay at each, and that delay is doubled from the caller's point of view because it applies to both directions. It is also the cheapest thing on this page to fix: terminate the published number where the agent runs. The same forwarding chain degrades audio quality, which is a separate failure diagnosed in the agent that is accurate at three and useless at seven.

The fixes, in order of payoff

1. Cache the catalog in memory with event-driven invalidation. Usually the largest single win, and it costs days rather than weeks.
2. Cut the endpoint wait, and add barge-in so the cost of cutting early is small. Configuration, measured in hours.
3. Stream everything. Start transcribing before the caller stops, start speaking before the full response exists, and never wait for a complete sentence to begin audio.
4. Remove turns. Every clarification is a full round trip, so where the confidence threshold sits is a latency decision as much as an accuracy one — the per-field thresholds in setting the point at which the agent stops and asks buy or spend whole seconds.
5. Shorten the call path and keep instances warm. Small, permanent, and mostly nobody's favourite work.
6. Only then look at the model. Changing where it runs and how it is served is a real lever — the trade-offs are set out in private LLM deployment — but it is the fifth thing to try, not the first.

What an abandoned turn costs, and what it is not

A caller who hangs up at turn 3 is not a partial order. Unless the build persists the in-progress order and the number, it is a missed call with the same four downstream outcomes as any other, set out in what happens to the calls nobody answered. The cheap mitigation is to persist every partial order with a callback number, so that at least the ones worth chasing can be chased.

It is also worth counting the turns you add for other reasons. Collecting payment during a call adds a round trip and a wait on the guest's phone, which is one of several reasons the shapes in taking payment on a phone order without a card number push the charge to collection or to a link. And for regulars, the fastest ordering channel is not a faster agent at all: a saved basket re-ordered in 2 taps skips the entire spoken menu, which is part of the case examined in whether anyone will install your restaurant app.

Latency is not a tuning exercise you do once. It is a budget with an owner, instrumented per stage and checked at peak — the kind of thing AI agents and automation work has to build in from the start, alongside the other decisions in voice and phone ordering and the wider restaurant and food service stack.

• voice ordering
• latency
• diagnostics
• catalog