AI agents vs workflows: one of our seven systems had to be an agent

The whole question is who owns the order of steps

Anthropic draws the line in one sentence each. From their engineering write-up on building effective agents: "Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage." Nothing in there is about how smart the system is or how many tools it has. The only difference is who decides what happens next.

The first gate is a question you can answer before writing any code. If the order of steps is the same every time, you want a pipeline. If the order changes based on what the model sees, you want an agent. Most projects fail that test in the direction nobody expects: the team describes an agent, then works out on the whiteboard that every run does the same five things in the same sequence.

The second gate is ours and it came out of production. What does a wrong answer cost, and can a machine detect it? If the wrongness is machine-checkable, meaning the code does not compile or the link returns a 404, a code gate beats agentness: cheaper, faster, and when it fires you know which check failed. Autonomy is for the cases where wrongness is a matter of judgement. The same test decides, one layer up, whether a process is worth automating at all.

Anthropic's own advice is to find "the simplest solution possible, and only increasing complexity when needed", and they are blunt that "optimizing single LLM calls with retrieval and in-context examples is usually enough". Dex Horthy puts the observational version in 12-Factor Agents: "most of the products out there billing themselves as 'AI Agents' are not all that agentic. A lot of them are mostly deterministic code, with LLM steps sprinkled in at just the right points." Agentness gets bought to avoid designing the system: if the model decides everything at runtime, nobody decides anything at design time, and the bill arrives later in a trace nobody can read.

AI agents vs workflows, one dimension at a time

Each row below is a trade you make on purpose. It is not a scorecard where one column wins.

None of that makes agents a bad idea. It makes agentness a purchase: you buy flexibility and pay in cost predictability and debuggability. Worth it when the flexibility is the product.

Seven systems in production, one honest agent

Definitions are cheap, so here is our own record, sorted by the first gate. All seven are running.

Two of the seven have no model in the hot path, and they are the two highest-volume systems we run: an OTT backend where tens of thousands of viewers press play inside the same minute, and a Telegram tracker doing millions of messages a day. At that volume a model call per item is a cost line and a latency line, and neither system has a question a query cannot answer.

One entry is arguable: the AI rector. Leadership asks it open-ended questions in plain language and it assigns tasks to departments, which sounds agentic twice over. But every number is retrieved at the moment of the question, a person confirms assignments before they go out, and every action is logged. The model chooses what to look up, code owns what happens next. By the first gate that is a workflow with a flexible front door, which is where internal assistants land once somebody asks who signed off on the write.

In the pipelines, the hard part was never the autonomy

Take the design-to-code pipeline. The tempting design is a reviewer agent that looks at the generated screen and decides what to fix. What we shipped re-renders the code headlessly and diffs it pixel by pixel against the original frame. Auto-layout edge cases were misaligning roughly 15% of frames, and a pixel diff measures that in milliseconds while a judgement call costs a round trip and is still an opinion. Failures go back into the queue with the diff attached. There is a loop, but code owns it: the retry is triggered by a number, not by a model's opinion.

The content pipeline is plainer. A term comes in from a CSV, the live sitemap is fetched so links point at pages that exist, one GPT-4o call writes the HTML body, every link gets a real HTTP request, a second call returns the SEO metadata as structured JSON, and the payload goes to the admin API. It replaces the 20-40 minutes of writer time each term used to cost, and the order has never needed to change, because the work was rule-bound before anyone automated it.

The 500K-record knowledge base is where people reach for an agent hardest, because the obvious framing is research: search, read, search again. Structure won instead. Retrieval resolves the entity to a canonical ID first and searches inside that constraint, hybrid plus reranking, and every number is validated against its source chunk before the answer goes out. The intelligence went into retrieval design, not into letting a model wander a large base and decide when it was done.

What made each of the three trustworthy was a check a machine could run. That is the second gate doing the work, and it is most of what we found when we went looking for where AI deployments actually break. None of those failures would have been fixed by giving the model more freedom.

The one that genuinely had to be an agent

The exception is a voice race engineer for live iRacing sessions. A driver holds push-to-talk mid-corner and asks something out loud. The answer has to come back in under two seconds, grounded in telemetry that keeps changing while the sentence is being spoken.

It fails the first gate outright. There are 19 MCP tools across six categories, from raw telemetry to pit commands. "Should I pit this lap?" pulls the current pit service status, the car's average lap time, the leaderboard and the damage report inside one reasoning loop. A question about tyre wear needs a different subset. You cannot write that order in advance, because you do not know the question in advance. The model picks the tools and the order from what it heard.

Agentness was not free, and the bill shows up in the design. Keeping the full tool-call chain in conversation history poisons the next answer: a lap time captured 30 seconds ago is already wrong, and to the model it is just more text. So history is a sliding window of four messages, every intermediate tool call runs fresh and is thrown away after, and live state lives outside the model in a worker that polls the sim every 100 ms. Most of the engineering went into constraining an agent that had to stay one. The wiring itself is the subject of connecting an agent to live systems through a tool layer.

Even here we took the control flow back where it mattered. Reading is autonomous. Pit commands pass through a human-in-the-loop gate: the agent recommends and prepares a stop, the driver confirms before anything reaches the car. The model can own the order of the reads. The writes are not its call.

Production agents are shorter than the demos suggest

The largest first-hand look at this is Measuring Agents in Production, built from 20 in-depth case studies and 86 surveyed practitioners across 26 domains. Its summary of how deployed agents are actually built: "68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation." Reliability, meaning consistent correct behaviour over time, is the top reported challenge, and the teams handle it through systems-level design rather than model choice.

Short chains dominate for arithmetic reasons. Reliability multiplies down a chain instead of averaging across it: at 95% per step, with the steps independent, twenty in a row come out right 35.8% of the time. The 95% is an assumption rather than a measurement, so substitute your own, but the curve keeps its shape.

A pipeline does not escape that arithmetic, it interrupts it. Every machine-checkable gate resets its term: once the link check has caught the 404, that step stops contributing to the product. Which is why the 68% reads as calibration rather than caution, and why step eleven usually costs more than it returns.

More agents is a shape decision, not an upgrade

Google Research ran the systematic version of the multi-agent question across 180 configurations, five architectures and four benchmarks. On sequential planning work, every multi-agent variant they tested degraded performance by 39-70% against a single agent. Independent agents working in parallel without talking to each other amplified errors by 17.2x. An orchestrator in the middle held the same amplification to 4.4x.

The other half keeps it honest. On parallelizable financial reasoning, centralized coordination improved performance by 80.9% over a single agent. So the question was never whether multi-agent is good. It is whether your task splits into pieces that can run at the same time without stepping on each other.

Cognition reached the same boundary from practice. Don't Build Multi-Agents, by Walden Yan in June 2025, sets out two principles - "Share context, and share full agent traces, not just individual messages" and "Actions carry implicit decisions, and conflicting decisions carry bad results" - and recommends a single-threaded linear agent as the default. Ten months later he published the revision: "multi-agent systems work best today when writes stay single-threaded and the additional agents contribute intelligence rather than actions." A sharpening rather than a retraction, and it lands where our own agent did. Extra intelligence, fine. Extra hands on the same object, no.

The best argument for agents, taken seriously

The strongest case against all of this deserves stating properly. Sean Goedecke argues in build agents, not pipelines that the most successful AI products, Claude Code and Codex and Cursor and Copilot, are agents and not by accident: "coding is a hard enough task that you simply cannot build a functional coding agent with pipelines." His second point is sharper: pre-assembling context fails because working out what information is relevant to a problem is often as hard as solving the problem, which is what broke naive RAG. The third should worry anyone who writes pipelines for a living. Agents inherit model improvements, so a new model can make an agent work on tasks that were out of reach a year ago, while a pipeline only gets incrementally better.

All three are true. We would add a fourth option the framing skips, and our counterexample happens to be a RAG system: when relevance is hard, autonomy is one answer and structure is another. On the 500K-record base, resolving the entity to a canonical ID before searching solved the relevance problem that embedding search could not, deterministically and at a fixed cost per query.

He concedes the boundary himself, and the concessions are the useful part: pipelines still win on predictable cost, tight context windows, and large volume. Our record puts numbers on those, because millions of messages a day and a two-second voice budget are not abstract constraints. His argument holds where relevance cannot be pre-computed, which is exactly the shape of the race engineer. So the disagreement is narrower than the titles suggest. We just find fewer of those tasks in the average business system than in a coding tool.

Five questions that classify your own project

Run this yourself, on a whiteboard, before anyone writes code or sends you a quote. Answer for the system you want in production, not for the demo of it.

Three or more answers in the left column and you are building a pipeline, whatever the deck calls it. That is a good outcome: it costs less to run, it breaks where you can find it, and you can still put a model in the two steps that need judgement. One layer above sits whether to build at all or buy something that already does it, worth settling first: a bought tool has an architecture too, and it is usually a pipeline.

The most common follow-up is the first row: how do I know whether my order is really fixed? Write down the last twenty real requests your process handled and mark the ones that would have needed a different sequence. If that number is zero, you have your answer. We build both, for what it is worth, and the pipeline is the one we quote more often. If you would rather have someone argue the other side against a specific project, a short audit does that in a call.

FAQ