Orders that sit in pending forever: tracing the transition that never fired

Three different failures present as a pending queue, and one query separates them before any triage begins. Group every order still in pending by age bucket — under 2 hours, 2 to 24 hours, 1 to 3 days, 3 to 14 days, older — and cross that with the last event type recorded against the order. The shape of the resulting grid names the population. A dense cluster in one age band all sharing a last event is an inbound event that stopped arriving at a moment in time. A flat spread across every band with 'created' as the last event is a human who was never told. A pile that is uniformly older than some date is a state with no outbound transition at all.

Payment failures are a separate population with their own diagnosis and are not covered here. Nor is the case where the order is fine and the availability underneath it was not — that symptom is traced in available in search and gone at checkout.

Split the queue before triaging any of it

This query needs a transitions table — a row per state change with its timestamp, its actor and its cause. If all you have is a status column, you can see that 340 orders are pending and nothing about why, and the investigation stalls immediately. That is the practical reason the argument in an event log or a status column for marketplace orders resolves the way it does: the projection is for reading, the log is for answering questions like this one.

Four reasons a transition never fires

1. The inbound event was dropped or never verified. A payment provider, a logistics partner or a seller's system was going to tell you something and the message did not land. The usual cause is dull: a rotated signing secret, an endpoint that started returning 500s during a deploy, or a signature check that fails silently and logs at debug level. Most providers retry on a backoff for a bounded window and then give up — read your provider's own documentation for its policy rather than assuming — so an outage longer than that window leaves a permanent hole that only a replay fills.
2. The notification was never sent, never delivered, or never seen. Three failures wearing one face. Never sent: no notification row exists, because the enqueue happened inside a transaction that rolled back, or behind a condition nobody tested. Never delivered: a row exists with no provider identifier, or with a bounce. Never seen: delivered to a push token that expired months ago, an address that forwards to a shared inbox, or a seller who has notifications muted. Each has a different fix and they are indistinguishable until you look at all three columns.
3. The state is terminal in practice. Somebody defined 'awaiting seller response' and built the way in without building the way out. The confirming query is straightforward: list every (from state, to state) pair observed in the last 90 days, then find any state that appears as a destination and never as an origin. Those states are where orders go to stop. This is also the point at which the difference between a true state and a flag becomes expensive, which is the distinction drawn in order states versus order flags.
4. The timeout was specified and never built. The requirements said sellers have 24 hours to accept. Somebody built the acceptance path, the reminder email and the countdown in the UI, and nobody built the transition that fires when the countdown reaches zero. The UI shows expired; the record says pending. Search the codebase for the timeout constant and see whether anything other than a template reads it.

The queue has to drain when nobody is at a desk

Split age-in-state by the hour and weekday an order entered the state. If the p95 for orders created after 18:00 on a Friday is 60 hours while the Tuesday-morning figure is 90 minutes, the state machine is not broken — it is staffed, and you have discovered that its throughput is a rota rather than a system. Reminders will not fix that. The two things that do are an automatic default at the deadline, and routing that does not depend on one person.

• Give every human-exited state a defined default. Auto-accept, auto-decline, auto-reassign or auto-refund — the business chooses, but something must happen without a person. A deadline with no default action is a reminder, and reminders decay into noise within about a month.
• Escalate to a role that is always occupied, not to an individual. The most common weekend failure is a queue routed to a named seller who is on holiday, with no second recipient anywhere in the configuration.
• Make the default visible before it fires. A seller who loses an order to auto-decline while asleep will forgive a system that warned them and will not forgive one that did not.
• Check whether the orders piling up were routed to the right seller in the first place. Unaccepted orders concentrated on a handful of sellers is an allocation problem rather than a notification problem, which is the territory of hand-tuned weights versus learned ranking for a small market.
• Treat the sweeper that applies defaults as a piece of production software with alerting, not as a cron line in a comment. Ours is the kind of unglamorous machinery we build as AI agents and automation when the decision at the deadline needs more than a fixed rule.

A state nobody can leave is not a state. It is a place the platform sends orders it has no plan for.

Replay, notify, or build the missing state

1. Does the last recorded event on a stuck order come from outside your system? If it does and no successor arrived, count inbound deliveries per day for the affected window against the sender's own log. A gap confirms lost events, and the repair is a replay from the sender plus verification and retry on the receiving endpoint.
2. If the last event is internal and a person was supposed to act, check for the notification row, then its provider identifier, then its delivery receipt. Missing row means the enqueue never happened. Row without identifier means the send failed. Delivered with no engagement across many orders means the channel is dead and needs replacing rather than retrying.
3. If a person did act and the order still did not move, the transition itself is the fault: a guard clause that never passes, a permission check that returns false for the actor, or an update that writes a column nothing reads.
4. If no transition exists at all out of that state, stop treating it as an incident. Design the exit: what ends this state, who may end it, what happens by default, and after how long.
5. Once the population is drained, add the standing check that would have caught it — a count of orders in each state older than that state's expected dwell time, evaluated hourly, alerting when it grows rather than when it is non-zero.

The permanent fix: states designed with their exits

The permanent fix is not a bigger dashboard. It is that every state in the machine carries a declared expected dwell time, an owner and a default. States that cannot answer those three questions are the ones that fill up, and they fill up quietly. That property — states designed with their exits rather than only their entrances — is what separates a marketplace whose ops team lives in a queue from one whose ops team lives in exceptions, and it is part of the wider argument about building software that is honest about its own failure modes set out in what an AI-native product studio actually is.

If the pending order is one of a pair fighting over the same supply, the diagnosis is different and starts elsewhere — see two confirmed bookings on one slot. The rest of this silo sits under marketplace architecture and the transaction data model, and the wider practice is described under marketplaces and two-sided platforms.

• order states
• webhooks
• queues
• incident response