AI agent observability: what to log before it breaks

Your dashboards stay green while the agent is wrong

Classic monitoring asks whether the request succeeded and how fast. For an agent that misses the point: the endpoint returns 200 OK on time while the answer quotes a refund policy you retired in spring, and nothing in APM knows what the right answer was.

Agents also loop, call the same tool four times with slightly different arguments, or announce that a task is done when it isn't. The researchers behind the MAST taxonomy annotated more than 1,600 traces from seven multi-agent frameworks and sorted the failures into 14 modes. One of their three top-level categories is task verification: the system finishes without checking that the job was actually done.

Most teams know this in principle. LangChain's State of Agent Engineering (1,340 responses, fielded in late 2025, published June 2026) found that 89% of respondents have some form of observability and 62% trace individual steps and tool calls, yet only 37.3% evaluate live traffic. It is a vendor survey (LangChain sells LangSmith), but the gap is real. The failure classes themselves (stale context, invented facts, folding at peak load) have their own post-mortem. This piece is about catching them.

In agent observability, the unit is the run, not the request

A run is one task from start to finish: the user's question, every model call and tool call the agent made to answer it, and the way it ended. In tracing terms the run is a trace and each step is a span nested inside it. Per-request logs show eight model calls; only the trace shows they were one confused attempt at a single refund.

The spans worth having are few: model calls, tool calls, retrieval, memory reads and writes, handoffs to another agent, and human approvals. Framework SDKs already think this way: the OpenAI Agents SDK wraps each Runner.run() in a trace and emits spans for agent runs, generations, function tool calls, guardrails, handoffs, and speech-to-text and text-to-speech audio.

On the root span, record agent build, prompt version, model id, tool schema version, a pseudonymous user id, total tokens and cost, and how the run ended, as one field with fixed values: success, turn limit, budget cap, tool error, abandoned. A run that ran out of turns and apologized nicely is still a failure.

One trap is easy to miss. Tracing SDKs ship spans in the background so the agent doesn't wait on uploads, and LangSmith's support team explains the "incomplete" status this produces: if the process exits before the upload finishes, the end event never arrives, and the fix is calling wait_for_all_tracers() before exit. So the crashed run you most need is the one most likely to be cut short: flush on shutdown and on every error path.

How do you monitor AI agents in production? Start with these signals

The minimum we would wire on any agent before real users touch it. Starting alerts are relative on purpose: set them from your own first two weeks of traffic.

If a week is all you have, start with run outcome, tool results with their arguments, and versions on the root. Together they turn "the agent was weird yesterday" into a query you can run.

Tool calls are where agents actually fail

A model call that goes wrong produces a bad paragraph. A tool call that goes wrong changes something: a ticket closed, an email sent, a record overwritten. Every tool span should carry the tool name, the arguments exactly as the model produced them, the validation result, the response status and the duration. Keep arguments even on success: the classic agent bug is a valid call with the wrong customer id.

Track error rates per tool and per class (bad arguments, timeout, auth, upstream 5xx): a healthy average can hide one tool failing since a partner rotated an API key.

Writes need two more fields. An idempotency key, so a retried run cannot send the same refund twice and you can prove it didn't. And an approval record when a human gate sits in front of the action: who approved, what exactly they were shown, the decision and how long it took. Log the approval as its own span, not as a note on the tool span. Approval time shows whether people review or rubber-stamp, and every rejection is a clean quality label.

Which tools may write at all is a permissions question, and connecting AI agents to your data covers the boundary: reading can be autonomous, writing goes through a gate. Observability is how you show the gate held.

Quality never shows up in the error rate

An answer can be wrong without anything erroring, and the only way to see it is to score it. Online, an evaluator (a rule, a model acting as judge, or a user's thumbs-down) scores a sample of live traces. Offline, a golden set of real questions with known good answers runs against every prompt or model change before it ships.

Sampling keeps the judge bill sane. HoneyHive's monitoring guide (updated 1 November 2026) suggests scoring all traffic under 1,000 events a day, 25-50% between 1,000 and 10,000, and 5-25% above that, while always capturing failed requests. That is one vendor's starting point, not a standard. Whatever the rate, keep failed and flagged runs at 100%.

Build the golden set from traces, not from imagination. Each time an eval or a user flags a run, copy the input and the corrected answer into the set. The specific checks for invented facts, such as groundedness per claim and a log line for empty retrieval, are laid out in how to catch AI hallucinations.

Watch the bill too: in LangSmith, an online evaluator that scores a trace while retention extension is enabled upgrades that trace to extended retention, which costs extra.

AI agent monitoring: alerts worth waking up for

When everything pages, nothing does. Sort alerts into tiers before writing the first rule.

Latency needs a budget per step, not one number for the whole run. Write down the end-to-end target and split it across the steps: speech-to-text, first model call, each tool, final model call and text-to-speech for a voice agent, or retrieval and model calls for a text one. Then alert on the step that overspends. Knowing the CRM lookup was slow is something you can fix before lunch; knowing the run was slow is not.

HoneyHive's guide says the thresholds are yours to set from your own baseline. The exception is the cost cap per run: put it in code on day one, then move it just above the longest legitimate run you observe.

Use the OpenTelemetry GenAI semantic conventions, and pin the version

The vendor market is moving fast, which is the strongest argument for instrumenting through OpenTelemetry. Langfuse is built on OpenTelemetry and accepts OTLP, and Arize Phoenix is built on OpenTelemetry and OpenInference, so OTel-instrumented code can switch backend with a config change rather than a rewrite.

Know what you are standardizing on, though. In release v1.42.0 on 12 June 2026, OpenTelemetry deprecated every gen_ai.* attribute, metric, event and span in the core conventions repo and moved them to a dedicated semantic-conventions-genai repository. On 13 November 2026 that repo shows no tagged releases, and its schema URL section still reads "TODO". The agent spans spec is marked Development and names operations such as invoke_agent, invoke_workflow and plan. Expect names to keep moving.

So pin it. Lock the instrumentation library version, record the convention version you emit as a resource attribute, and treat an upgrade like a schema migration: check dashboards and alert queries against the new names before you deploy. Note that message content (gen_ai.input.messages, gen_ai.output.messages) is Opt-In, with the warning that it is "likely to contain sensitive information including user/PII data". The standard keeps it off by default for a reason.

Your traces are a copy of your customers' data

A support agent's trace holds the customer's message, their order history and the answer. That is usually personal data under the GDPR, so your observability vendor is a processor that needs a DPA, a region and a retention policy, like your CRM.

The defaults lean the other way. In the OpenAI Agents SDK tracing is on by default and trace_include_sensitive_data defaults to True, which stores the inputs and outputs of model generations and function calls. Audio spans include base64-encoded PCM of the input and output audio unless you switch off trace_include_sensitive_audio_data, so a voice agent can end up with recordings of people's voices in its trace store.

Most of the fix is configuration. Mask in your own process, before export. Langfuse's SDKs take a masking function (mask_otel_spans in Python, mask on the LangfuseSpanProcessor in JS/TS), and its masking docs warn that collector-side masking only happens after data has left the application. LangSmith offers `LANGSMITH_HIDE_INPUTS`, `LANGSMITH_HIDE_OUTPUTS` and a regex anonymizer that also run client-side. Attach a pseudonymous user id to every trace so a deletion request is one query; Langfuse documents deletion through its API, usually complete within 15 minutes.

Choose the region when you sign up, because neither Langfuse nor LangSmith migrates data between regions later. LangSmith's EU region runs on Google Cloud, but LangChain has no EU legal entity for contracting.

Then write down a retention number. If the agent counts as a high-risk system under the EU AI Act, Article 19 and Article 26(6) require keeping its automatic logs for at least six months, unless other law, data-protection law in particular, provides otherwise, from 2 December 2027 for Annex III systems. Most business agents are not high-risk; the AI Act guide helps you check.

What tracing costs: five vendors, five billing units

Each vendor bills a different thing. List prices below were retrieved on 13 November 2026; check the vendor page before you budget.

A worked month shows why the unit matters. Assume 1,000 runs a day, so 30,000 a month, and each run is one trace with five model calls, eight tool calls, one retrieval and one eval score. This is list-price arithmetic, not a measurement.

Langfuse Core: 16 units per run, 480,000 in the month. $29 plus 380,000 extra units at $8 per 100k comes to about $59, with 90 days of access.

Datadog Pro, annual terms: only the five model calls bill, so 150,000 LLM spans. $160 plus five blocks of 10k at $3.50 is about $178. Tool spans are free here and billed everywhere else.

LangSmith Plus, one seat: 30,000 traces, 20,000 above the included 10,000. The overage rate is not on the public pricing page today, so the figure is $39 plus whatever your billing settings show.

Braintrust and Arize AX: both depend on payload, and Arize does not list overage. At 20 KB per run the month is 0.6 GB, inside Braintrust's free gigabyte; if each run drags 200 KB of retrieved context, it is 6 GB and past Pro's 5 GB. Arize AX Pro includes 50k spans; this month produces 450,000.

W&B Weave also bills by data volume: its pricing page, now on CoreWeave's site, lists 1 GB a month free and Pro from $60 with 1.5 GB included, then $0.10 per MB.

Picking LLM observability tools while the market consolidates

The company running your tracing tool today may not be running it next year. ClickHouse took over Langfuse in January 2026, promising no licensing changes and self-hosting as a first-class path. Mintlify acquired Helicone in March and put it in maintenance mode, offering to help customers move elsewhere. Cisco completed its purchase of Galileo in May and is folding it into Splunk Observability. We would still use a vendor. We would just keep the instrumentation portable enough to leave.

For LangSmith alternatives, Langfuse vs LangSmith comes down to hosting more than features: Langfuse is open source and self-hostable, while LangSmith is cloud-first, with hybrid and self-hosted deployment only on Enterprise.

Self-hosted vs SaaS is mostly a question of who restarts ClickHouse at 3 a.m. Langfuse's self-hosting guide lists web and worker containers plus PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible storage. If nobody on the team runs that stack already, an EU cloud region with a DPA is often the safer privacy choice. If a contract says traces never leave your infrastructure, self-host and budget the ops time.

What to wire this week

None of this needs a platform decision first. On an agent that already runs, a week of work looks like this:

1. Give every run a trace id and an outcome field. Set it in code on every exit path, errors and turn limits included.

2. Put versions on the root span. Without them a regression and random noise look the same.

3. Log every tool call with its arguments and status. Add an idempotency key on writes and a separate span for each human approval.

4. Flush on shutdown and on errors. Then kill the process mid-run on purpose and check that the trace arrived.

5. Mask before export and fix the region. Switch off default raw content capture, attach a pseudonymous user id, write down the retention period.

6. Start a golden set of 20 real questions. Pull them from traces and run them before every prompt or model change.

7. Write a latency budget per step and a cost cap per run. Alert on the step, not the total.

8. Price your own month. Count spans and payload per run from one day of traces and run them through the table above.

One lesson from our own work. In the Race Engineer, a voice agent that answers an iRacing driver in under two seconds using 19 MCP tools, the agent deliberately keeps only the driver's question and its final answer in context (MAX_HISTORY = 4). Every intermediate tool call runs fresh and is then discarded, because a lap time from 30 seconds ago is already wrong. Right for answers, but it means the conversation history is not a record of what the agent did: tool calls dropped from context exist only if something logs them separately, and the same goes for the driver's confirmation at the human gate on pit commands. If your agent trims its memory, make sure your logs don't.

And if you'd like a second pair of eyes on what your agent records before it goes live, that is part of our free audit.

FAQ