AI agents in production: what actually breaks

Building an agent that completes a task in a demo takes an afternoon. Building one that completes the same task ten thousand times unattended, against systems that occasionally return nonsense, without doing anything expensive or irreversible when it misunderstands, is a different job.

The failure modes are consistent enough to enumerate.

1. Error compounds across steps

This is the arithmetic that kills most ambitious agent designs. If each step of a plan is independently 95% reliable, a five-step plan is about 77% reliable end to end, and a ten-step plan is about 60%. Reliability that feels excellent per step is unusable per task.

The containment pattern is to shorten the chain rather than to improve the model. Most "20-step agent" designs are better expressed as conventional code that calls a model at the three points where judgement is genuinely required. Deterministic control flow, probabilistic decisions inside it.

Use an agent where the sequence of steps genuinely cannot be known in advance. Everywhere else, write the workflow and let the model make the judgement calls inside it.

2. Tool calls that succeed technically and fail semantically

The model calls your search tool with a plausible but wrong query, gets zero results, and confidently concludes the record does not exist. Nothing errored. Every log line is green. The answer is wrong.

Containment is mostly tool design rather than prompting:

• Make tools return informative failures. "No results for query X; 4,182 records exist in this index" is far more recoverable than an empty array.
• Constrain arguments with enums and schemas rather than free text wherever the domain allows it.
• Prefer fewer, well-named tools. Twenty overlapping tools produce mis-selection; five clear ones do not.
• Return the tool result verbatim rather than a summary. Summarising before the model sees it discards exactly the detail it needed.

3. Loops, and the bill that comes with them

An agent that cannot complete a task will often keep trying — rephrasing the same failing query, re-reading the same document, oscillating between two approaches. In a demo you notice and stop it. Unattended at 3am it runs until something else stops it.

1. Hard caps on steps, wall-clock time, and spend per task. Not advisory limits — enforced ones.
2. Loop detection: if the last three tool calls are materially identical, stop and escalate.
3. A budget that is per-task, not per-tenant, so one pathological input cannot consume the day's allowance.
4. An explicit "I cannot do this" terminal state that is treated as a valid outcome rather than a failure to retry.

4. Side effects without idempotency

The moment an agent can write — send an email, create a ticket, post a transaction, update a record — retries stop being free. A timeout that triggers a retry after the action already succeeded produces duplicates, and duplicates in a financial or clinical system are a serious incident rather than a bug.

• Every write tool takes an idempotency key derived from the task, not generated per attempt.
• Separate read tools from write tools explicitly, and make write tools the small, heavily reviewed minority.
• Require confirmation for actions that are irreversible or externally visible — sending, publishing, paying, deleting.
• Log the intent before the action, not after, so a crash mid-write is diagnosable.

5. The agent acts with more authority than the user

A common and dangerous default: the agent holds a service account with broad access, and acts on behalf of whichever user asked. Now anyone who can talk to the agent can, with sufficiently creative phrasing, reach anything the service account can reach.

The fix is to propagate the calling user's identity through every tool call and let the underlying systems enforce their own permissions. This is more work and it is not optional. It also means your retrieval layer has to be permission-aware — an index built without access-control filtering will happily surface documents the requester was never entitled to see.

6. Untrusted content becomes instructions

If an agent reads web pages, emails, tickets, or uploaded documents, that content can contain text addressed to the agent. Treating retrieved content as instruction rather than data is the central security problem of agentic systems, and there is no prompt that fully solves it.

• Keep the privilege boundary outside the model: an agent that reads untrusted content should not also hold the ability to take consequential actions unattended.
• Require human confirmation for consequential actions in any flow that touches external content.
• Mark retrieved content clearly as data in the context, and never let it modify the tool set.
• Assume containment, not prevention, and design the blast radius accordingly.

7. You cannot see what it did

When a user says the agent did something strange last Tuesday, you need the full trace: inputs, retrieved context, each tool call with arguments and results, the model version, and the final output. Without it every complaint is unfalsifiable and every fix is a guess.

Trace storage feels expensive until the first incident, at which point it is the only thing that matters. Getting from a working agent to one you can operate is the same gap we describe in proof of concept to production.

The short version

1. Shorten the chain. Deterministic workflow, model judgement at the few points that need it.
2. Design tools to fail informatively, and keep the set small.
3. Enforce hard caps on steps, time, and spend; detect loops; allow giving up.
4. Make every write idempotent and confirm the irreversible ones.
5. Propagate the user's identity; make retrieval permission-aware.
6. Never let an agent that reads untrusted content also act unattended.
7. Trace everything.

• AI agents
• engineering
• production
• reliability