Why AI agents fail in production - a post-mortem with real numbers

Ask why AI agents fail in production - the answer is rarely loud

The picture people carry of a failed AI system is dramatic: it hallucinates something absurd, someone screenshots it, the project dies. Real production failures are quieter than that. The model produces something plausible, a person skims it, it ships, and the damage surfaces two weeks later in a support queue or a broken page nobody traced back to the launch.

There is a reason stronger models make this worse, not better. As Atlan puts it in their write-up on the same problem, a weak model produces obvious errors and a strong model produces convincing ones. A bad answer you can see is cheap to catch. A bad answer dressed as a good one is the expensive kind, because it clears review on fluency alone.

Almost every failure below has the same root. The team trusted the output because it read well, instead of grounding it in real data and checking it before it left the building. Or, in the words of a Towards Data Science piece on agents built backwards, good models do not save bad architecture. The fix is never a smarter prompt. It is grounding plus a gate.

Plausible code that passes the demo and fails in review

A Figma file is not a specification. It is a set of visual hints: buttons with no semantic names, spacing tokens in a separate library, auto-layout rules that mean different things in different frames. Point a generic model at that ambiguity and it fills the gaps by guessing. On a design-to-code pipeline we run, the generic version hallucinated class names and invented props that did not exist - and the output looked plausible enough to pass a glance.

That is the trap. Code that looks right and renders wrong on the first run is worse than no code, because someone has to notice it is broken before they can fix it, and by then it is in the branch. Nested components made it sharper: when a card holds a badge that holds an icon, the context window fills with parent-component noise, and the model starts inventing prop names to bridge what it can no longer see clearly.

Two gates fixed it, and neither is clever. First, a structural check with Pydantic: every component name, prop and import has to resolve before a single file is written, so structurally broken code never leaves the pipeline. Second, a visual gate: auto-layout edge cases were misaligning roughly 15% of frames, so the generated code is re-rendered headlessly and diffed pixel by pixel against the original frame. Failures go back in the queue with the diff attached, so the model corrects against the exact discrepancy instead of guessing again. The demo needed none of this. Production needed all of it.

The data was true a minute ago - that is the problem

Some failures are not about wrong facts but about old ones. We built a voice race engineer for live iRacing sessions that has to answer a driver in under two seconds, reading live telemetry through 19 tools. The obvious design keeps the full tool-call history in the conversation, the way a chatbot keeps its transcript. That design poisons the next answer.

A lap time captured 30 seconds ago is already wrong when the next question arrives, but the model cannot tell a stale number from a fresh one - it is all just text in the context. So it reasons, confidently, from data that expired mid-conversation. Stale data carried forward is worse than no data, because no data at least forces a fresh read.

The fix was to stop treating history as a data store. Only the driver's question and the final answer are kept - a window of four messages - while every intermediate tool call runs fresh each turn and is thrown away after. Live state lives outside the model entirely: a worker polls the sim every 100 ms and writes completed laps to a database the agent queries on demand. The rule that generalizes past racing: anything time-sensitive gets retrieved at the moment of the question, never remembered from an earlier turn.

The model makes up a number, and it looks real

The most dangerous thing a language model produces is a specific, confident fact it pulled from memory instead of from your data. On a RAG assistant over a 500K-record knowledge base, the failure was exact: ask for a drop rate and the model would state a precise percentage, confidently and precisely wrong. A vague answer invites a second look. A precise one does not, which is what makes it worse.

The fix is per-claim validation. Every number in the answer - a percentage, a timer, a required level - is checked against the source chunk it supposedly came from before the response is returned. A claim with no retrievable source is dropped or flagged as unverified, never smoothed over into fluent prose. This is not a gaming problem. Any company sitting on a large structured base - a product catalog, a spec sheet, a policy manual - hits the same failure the moment it puts a chat box in front of it: fluent answers that are subtly wrong on exactly the numbers people will act on.

The same instinct invents references, not just numbers. On a content pipeline that publishes SEO articles, the model wrote plausible internal links to pages that did not exist. Left alone, that fills a site with dead links and quietly erodes both SEO and reader trust - a failure invisible in a spot-check of three articles and obvious across three hundred. The guardrail runs in two places: before generation, the live sitemap is fetched and the real URLs are injected into the prompt, so the model links to pages that exist instead of guessing; after generation, every link gets a live HTTP request, and anything that returns a 404 or points outside the known sitemap is unwrapped - the anchor text stays, the broken link goes. The pattern under both cases is one sentence. If the model can state it, make it prove it against a source.

It works at average load and folds at the peak

The last failure mode is the meanest, because it is the one a demo structurally cannot show. A system can be comfortable at average load and fall apart at the peak, and the peak is usually the only moment that matters. We saw it on an OTT streaming backend: a live match kicks off and tens of thousands of viewers press play inside the same minute. A VOD title can stall for a second and nobody notices. A live football match cannot.

A naive setup fails as a cascade, not a single point. Authorization falls through to the database, the database saturates, timeouts back up into the services, and the slowdown spreads until the whole play path degrades at once - precisely during the event everyone tuned in for. The fix is architectural: Go on the latency-critical paths, Redis in front of the hottest data so authorization never hits the database under load, and event streams instead of blocking calls so one slow consumer downstream cannot stall the request in front of a viewer.

Scale hides the same way in smaller systems. A Telegram feed bot we run looks trivial - poll a feed, forward new items - until one posting matches hundreds of users at once and one job becomes hundreds of messages, all due within seconds and all under a rate limit. The answer there was a queue that absorbs the burst and workers that drain it at a controlled rate, requeuing on a 429 rather than dropping the message. Millions of messages a day, and none of it shows up in a test with three users. The load only exists in production, so the load has to be tested before production.

What the demo hides, put in one table

Five failures, one shape. Each looked fine in a controlled demo and broke on contact with real conditions: genuine ambiguity, time that keeps moving, live data, and a real load spike. Read side by side, the guardrail column is the useful part: none of it is a better model, all of it is grounding and gating around the model.

The column that never changes is the middle one. Production is where ambiguity, time, real data and load all arrive at once, and a demo removes all four by design. That is the whole reason a passing demo predicts so little.

A pre-deployment checklist you can run this week

You do not need us or a framework to pressure-test a deployment before it ships, and it does not matter whether you built the system or bought it - a purchased tool hallucinates the same way a custom one does. Run it against the checklist below. Each row is one of the failures above, turned into a question you can answer with a small experiment rather than an opinion.

Run these before launch and most of the failures in this post never reach a user. Skip them and you get the quiet kind of failure, the one that ships clean and surfaces weeks later. (This is the work we do day to day, for what it is worth - but the checklist stands on its own, whoever runs it.)

If you want a second pair of eyes on where a specific deployment is likely to break, a short audit is a low-key way to get one: you leave with the two or three failure modes most likely to bite your setup, mapped to the checks above.

FAQ