Voice Agent Latency: Where the 2 Seconds Actually Go

What latency should a voice agent target? Humans set the bar

Start with the boundary, because most arguments about voice latency are really arguments about where the stopwatch starts. The number that matters runs from the last sound the user makes to the first audible sample of the reply. Model inference is a slice of that. So is deciding the person has finished, transcribing them, looking something up, synthesizing speech and moving audio across the network.

The human reference point is well measured. Stivers et al. in PNAS (2009) timed question-answer turns in 10 languages and found a mean response offset of +208 ms, with a median of +100 ms. Japanese speakers averaged +7 ms and Danish speakers +469 ms, so the bar shifts with culture, but everywhere it sits well under half a second.

Machines are not held to that yet. ElevenLabs' latency guide says a natural-feeling agent typically targets a P50 time-to-first-audio under 800 ms and a P95 under 1.5 seconds. Twilio's November 2025 benchmarks put the mouth-to-ear turn gap at 1,115 ms target and 1,400 ms upper limit. The Voice AI & Voice Agents primer calls 1,500 ms voice-to-voice “an important target”. And Dasha says plainly that there is no source-backed universal threshold for “good”.

A workable rule sits inside those numbers: the typical turn under a second, and the slow tail, the turns people remember, under two seconds even when a tool call is involved.

Where the time goes in a voice agent architecture: a budget to fill in

A cascaded voice agent architecture is a chain: audio comes in, something decides the turn is over, speech-to-text produces a transcript, the LLM answers (sometimes after calling tools), text-to-speech turns the answer into audio, and the network or phone line carries it back. Streaming lets the stages overlap. The LLM hands sentences to TTS as they form, so you pay time to first token and time to first audio rather than the full generation.

The table puts three public sets of reference figures side by side. They are not measurements of your system, and they are not ours. The last column is for you.

Read the columns down, never across. Each source draws its stage boundaries differently, so a Twilio row plus an ElevenLabs row adds up to nothing. ElevenLabs calls its figures illustrative, Twilio calls its own “starting benchmarks, not best-in-class”, and Cresta's are ranges its team has seen. Notice too that two of the three say nothing about tool calls, which is exactly where real agents go over budget.

Turn detection: knowing when someone finished is the most expensive stage

Before anything else can run, the system has to decide the person is done. The simplest endpointing waits for silence, and Cresta's CTO writes that in realistic conversations a silence threshold of 600 ms is the bare minimum. Go shorter and the agent cuts people off when they pause to think. That 600 ms is spent before transcription even finishes.

Semantic turn detection reads the words and their rhythm instead of waiting for quiet. Cresta reports it can bring response times under 300 ms in common cases without cutting users off. Deepgram's Flux builds turn detection into the speech model itself, and its own figures (vendor self-reported, October 2025) are an end-of-turn signal within 1 second at p90 and 1.5 seconds at p95, 200-600 ms faster agent responses than pipeline approaches and about 30% fewer false interruptions.

Flux also offers an eager end-of-turn event that fires 150-250 ms earlier, at the cost of 50-70% more LLM calls. That trade sums up this stage. You buy speed by guessing early, and every early guess costs tokens and sometimes a reply you have to throw away. A p95 of 1.5 seconds on this stage alone is roughly the size of a whole-turn target in the budgets above, which is why tuning it on recordings of your own users beats any setting from a quickstart.

Push-to-talk deletes that stage, where you can use it

If the user tells the system when they have finished, there is nothing to detect. ElevenLabs' guide recommends manual commit for exactly this case, when the application knows the turn is over from another signal such as a push-to-talk release or a UI event. There is no silence timer to wait out, a cough or a colleague talking nearby cannot trigger a false barge-in, and transcription starts the instant the button comes up.

It fits users who are operators rather than callers: warehouse and field staff on a headset, dispatchers, technicians reading values back to a system, drivers, internal tools that one person uses dozens of times a day. They learn the button in an afternoon. Noisy places benefit most, because the button does not care about the forklift behind you.

It does not fit an inbound phone line. A customer calling your support number will not hold a key, and you cannot ask them to. The same goes for anything consumer-facing that has to feel like a conversation. There you pay for turn detection and tune it. The practical point is to decide who your user is before choosing the architecture, because for staff, push-to-talk is a legitimate design choice and not a compromise.

Tool calls are where the 2-second budget dies

A voice agent that only talks is fast. One that checks an order, a calendar slot or a stock level pays far more, and the reason is structural. Cresta puts it as a rule: a tool lookup before any message turns one LLM first-token latency into one full LLM latency plus one first-token latency. The model has to finish writing the tool call before anything runs, then the tool runs, then the model starts again with the result.

The primer's worked example makes it concrete: with a time to first token of 450 ms and 100 tokens per second, a 100-token function-call request takes 1,450 ms before the function even executes. Carry that through one realistic turn and you get the bars below.

That is about 3.1 seconds before you add turn detection and the network, for one lookup. Four lookups done one after another, each in its own model pass, would repeat the first two bars four times. So the levers are about the shape of the loop more than the speed of the model.

Start by running independent calls in parallel. If the model emits several tool calls in one response and your runtime executes them concurrently, you pay one model pass plus the slowest tool. Check both halves, because a naive loop awaits them one by one even when the model asked for them together. Then fetch what you can predict: Cresta's advice is to fire the calls you know you will need concurrently at the start of the call, and customer lookup by phone number is the classic case. Finally, cap the loop. LiveKit's latency guide rates function tools and MCP use as “potentially high” impact and points to a max_tool_steps limit.

While a tool runs, fill the silence honestly. LiveKit also suggests a thinking sound during tool execution and telling the user before the call is made. A short “let me check that” changes the gap the caller feels, but not the one you measure, so track both. And remember that the p95 of your CRM is now part of the p95 of your voice agent. How those systems get exposed as tools in the first place is its own topic, covered in connecting AI agents to your data.

Stale context costs latency and correctness

On every turn the model re-reads the conversation. Longer history means more input before the first token, and LiveKit notes that concise system instructions and trimming or summarizing older turns can reduce time to first token. The less obvious cost is correctness. In an agent that reads live data, an old tool result sitting in history looks exactly like a fact.

We ran into this building a voice race engineer for iRacing. The driver speaks over push-to-talk, Whisper transcribes, an OpenAI agent with 19 MCP tools reads live telemetry, and TTS answers in F1-radio style, in under 2 seconds including the tool calls. “Should I pit this lap?” fires four tools in one reasoning loop: pit service status, average lap time, leaderboard and damage report. Keeping the full tool-call chain in history poisoned the next question, because the agent reasoned from a lap time captured 30 seconds earlier that was already wrong.

The fix was a sliding window. Only the driver's question and the agent's final answer persist (MAX_HISTORY = 4), the system prompt stays pinned first, and every intermediate tool call runs fresh each turn and is then discarded. Lap history for all 64 cars lives in SQLite, written by a worker that polls every 100 ms, outside the model's context. The model gets less to read and nothing old to trust.

To be clear about what that figure does not cover: push-to-talk means no end-of-turn detection, there is no telephony leg, one trained user, no concurrent calls. It is evidence for the tool-and-context part of the budget, not a phone-grade number. The pattern transfers directly, though. Order status, stock levels and account balances change mid-call too, so keep the conversation and re-fetch the facts.

Speech-to-speech vs cascaded: what you trade for the lower number

A speech-to-speech model hears audio and answers in audio, skipping the transcription and synthesis hops. Fewer hops sounds like less latency, and for small talk it often is. LiveKit rates the choice between a pipeline and a realtime model as “none to medium” impact on latency, depending on your pipeline. It rates where the agent runs relative to its models as “very high”.

What you give up is control. Deepgram's comparison (worth reading knowing Deepgram sells cascade components) lays it out: a cascade produces text at every stage, which gives you an audit trail and lets you swap one component without touching the rest, while speech-to-speech needs a parallel transcription layer that can drift from what was actually said. It also questions whether tool use stays as consistent with audio in and out, and notes that observed costs can reach 4x theoretical minimums because token pricing grows with conversation length.

The bigger number is accuracy. The τ-Voice paper (March 2026) ran 278 customer-service tasks: a reasoning text agent passed 85%, voice agents 31-51% on clean audio and 26-38% with realistic noise and accents. Sierra's May 2026 update shows the best voice result climbing from 30% to 67%, about 79% of text capability. The gap is closing quickly, and it is still there.

Shaving 300 ms off a turn means little if the task completes two times in three. For agents that act on business systems, a cascade with streaming at every boundary is the safer default, and speech-to-speech fits jobs that are mostly conversation rather than transactions.

Voice AI cost per minute, from official price lists

Only official pricing pages, all retrieved on November 13, 2026. These move often, so re-check before you build a business case on them.

Two notes. OpenAI prices realtime audio per token and does not state how many audio tokens a minute contains, so there is no per-minute speech-to-speech figure here. Anyone quoting one did a conversion OpenAI does not publish. And latency has a price in both directions: ElevenLabs Flash voices cost $0.04 per minute on Retell against $0.015 for the platform voices, and Flux's eager end-of-turn buys its 150-250 ms with 50-70% more LLM calls. Whether those minutes pay back is a separate calculation, done properly in AI customer support ROI.

Measure voice agent latency for the whole turn at p95, not the model at p50

ElevenLabs quotes its Flash v2.5 model at about 75 ms of inference for short inputs and says plainly that the figure is inference only. Vapi's Humanness Index lists the same model at 197 ms time to first audio in its standings, where the benchmark method is a median of live streaming trials that includes network round trip. Neither number is wrong. They measure different boundaries, and only one of them is something a caller can hear.

So measure from the caller's side. Cresta's method is a good default: place a simulated call through your real telephony or WebRTC path, record it from the caller's end, and run an ASR model with word timestamps to compute the gap between the user's last word and the agent's first. Then look at percentiles, not averages. A mean of 900 ms can hide a tenth of turns at three seconds, and those are the ones people describe as “it froze”.

Split the numbers by turn type too. A plain answer, a turn with one tool call and a turn where the user interrupted have different budgets, and blending them makes every one of them look fine.

Acting by voice needs a confirmation gate

Reading data by voice is low risk. Booking, cancelling and refunding are not, and voice adds failure modes text lacks: mishearing, and interruptions mid-action. AAIF's production write-up (October 2026) notes that in LiveKit tool code continues running by default after the agent is interrupted, while Pipecat cancels in-flight function calls by default. Two lines from it belong on every team's wall: “A timeout cannot safely be treated as proof that nothing happened” and “Retrying without checking can produce two bookings.”

The pattern is simple: read autonomously, act with confirmation. The agent reads the action back (“Cancel Friday's 3 pm booking, yes?”), waits for an explicit yes, sends the write with an idempotency key, and only says “done” after the system confirms. That costs one extra turn, which is cheap next to a double booking. Count it as a deliberate turn rather than latency to optimize away.

If your callers are in the EU there is one more line of code to write. Article 50 of the AI Act requires systems that talk to people to be designed so they are told they are interacting with AI, unless that is obvious, at the latest at the first interaction (Art. 50(1) and 50(5)). These obligations apply from August 2, 2026 per the Commission's guidelines notice. For a voice agent that means the first sentence. The wider calendar is in our EU AI Act compliance guide.

A pre-launch latency checklist you can run alone

None of this needs a vendor. One engineer and a week of recordings are enough to run every row.

If you do one thing this week, make it the last row. Fifty recorded calls through your real phone path will tell you more about your voice agent latency than any vendor table, this one included.

The tool-calling end of this, including a confirmation gate on actions, is written up in the race engineer case. And if a second pair of eyes on your budget would help, that is what the free audit is for.

FAQ