Evaluating a Voice Agent Before Launch: Latency Budgets, Interruptions, Eval Sets and Simulation
November 4, 2026 · 7 min read · Ortavox Engineering
A practical pre-launch checklist for voice agents: how to measure the latency that matters, test interruption handling, build an evaluation set from real calls, run simulated callers, and decide what to automate versus listen to by hand.
A voice agent that passes every text-based test can still fail on the phone. It can be right and too slow, polite and unable to stop talking when interrupted, accurate in the demo and lost when the caller has a cold and a noisy kitchen. This guide is the checklist we use before an agent takes production traffic. It is tool-agnostic; the Ortavox-specific parts are called out.
1. Measure latency where the user feels it
The number that matters is the gap between the caller finishing a sentence and hearing the first syllable of the reply. In a cascaded pipeline on LiveKit Agents it decomposes cleanly: end-of-utterance delay (how long after the caller stops the agent decides they are done), LLM time to first token, and TTS time to first byte. LiveKit's metrics documentation defines total response latency as exactly that sum, and the SDK emits each component per turn.
Component | Typical healthy range | If it is high |
|---|---|---|
End-of-utterance delay | 200 to 500 ms | Turn detector too conservative, VAD silence threshold too long, or the model waiting on a trailing partial transcript |
STT finalisation | 80 to 150 ms after speech ends | Provider region far from the media server, or a batch-oriented model used in streaming |
LLM time to first token | 200 to 400 ms | Large model, long system prompt without caching, cold tool-schema parsing, or a slow retrieval call before generation |
TTS time to first byte | 80 to 250 ms | High-quality tier chosen where a low-latency tier would do, or sentence-level instead of streaming synthesis |
Network and buffering | 40 to 80 ms | Caller on mobile, SIP carrier hop, or jitter buffer sized for packet loss |
Report p50 and p95, per turn, over at least a few hundred turns. A p50 of 600 ms with a p95 of 2.5 s is a worse product than a flat 800 ms, because callers remember the stalls. In the Ortavox console every call has a turn-by-turn latency waterfall with these components, which is the fastest way to see whether a regression came from a prompt change, a model swap or a provider.
Measure with the voice you will ship. Switching TTS tiers for a demo and back for production is the most common reason a launch feels slower than the test.
2. Test interruptions on purpose
Interruptions are where most agents fail and where most test suites have no coverage. Build a scripted set of barge-in cases and run each one at least ten times, because the behaviour is timing-dependent:
- Early barge-in: caller speaks 300 ms into the agent's reply. Expect the agent to stop within a few hundred milliseconds and the transcript to show the reply as truncated, not as fully spoken.
- Back-channel: caller says "mm-hm" or "okay" while the agent is mid-sentence. Expect the agent to keep talking. A system that stops on every acknowledgement feels broken.
- False start: caller begins, stops, restarts. Expect one turn, not two, and no reply to the fragment.
- Overlapping end: caller starts speaking exactly as the agent finishes. Expect the agent not to re-open the floor with a filler like "anything else?".
- Noise: TV, kitchen, street. Expect no spurious interruptions. This is where noise cancellation and VAD thresholds earn their keep.
Check two things after each case: that audio actually stopped (listen; do not trust the log), and that the conversation state is right, meaning the LLM context contains what the caller heard, not the whole reply the agent intended to say.
3. Build an eval set from real calls, then keep it
An evaluation set is a fixed list of conversations with expected outcomes that you run every time anything changes. For voice, build it in three layers:
- Transcript-level cases, from the first 50 to 100 real or pilot calls. For each, record the intent, the required tool calls with arguments, the facts the agent must state, and the facts it must not state. These run fast and catch prompt regressions.
- Audio-level cases, a smaller set with the actual recordings: accents, numbers and names, spelling out an email, a child in the background. These exercise STT and turn-taking, which transcript tests cannot.
- Policy cases: the caller asks for something out of scope, requests a human, gives contradictory information, or tries to extract the system prompt. Expected behaviour is a specific escalation path, not a clever answer.
Score with a rubric that separates task completion, factual accuracy, tool correctness, tone and latency, and resist collapsing them to one number early. An LLM-as-judge can grade the first three reliably if you give it the expected outcome; it is weak at tone and useless at latency. Keep the set under version control and add a case for every production incident.
4. Simulate callers at volume
Simulation means another model plays the caller, over real audio, through your real telephony or WebRTC path, hundreds of times. It is the only way to find load-dependent problems before launch: concurrency limits, provider rate limits, queueing behaviour and the p95 you will never hit by hand.
A number of platforms now specialise in this. Cekura, Coval and Hamming each generate personas and scenarios, place calls and score the results, with different emphases: low-cost self-serve scenario generation, simulation variety with CI integration, and audio-level analysis with IVR emulation respectively. A peer-reviewed framework for judging such platforms (arXiv 2511.04133) measured both the realism of generated conversations and the accuracy of their automatic grading, and found statistically significant gaps between products on both; the authors' point is that the testers need testing too.
Whichever tool you use, three rules hold. Run simulations through the production entry point, including the phone number, so carrier and SIP behaviour is in scope. Mix personas: a terse caller, a rambling one, a non-native speaker, an angry one. And treat simulated scores as a regression signal, not a quality guarantee: real callers will find a pattern the simulator did not generate.
5. Load and failure drills
- Concurrency: push to your plan's limit plus one and confirm the extra call queues or fails with a clear message rather than connecting to silence. On Ortavox the pay-as-you-go tier includes four concurrent calls and queues beyond that.
- Provider failure: revoke a key or point at a dead region and confirm the call fails fast with the provider named in the log. If you rely on BYOK, test an expired key specifically.
- Long calls: a 30-minute conversation should not drift in latency or memory. Watch the waterfall over time, not just the first turn.
- Hang-up handling: caller drops mid-tool-call. Confirm the tool completes or rolls back and the call record is consistent.
6. Listen to calls by hand, on a schedule
Automation finds regressions; people find the problems you did not think to test. Before launch, have someone who was not on the project listen to twenty full calls with the transcript open. After launch, keep a fixed weekly quota, five to ten calls sampled across outcomes, and write down what they heard. Call replay with a synchronised transcript and per-turn latency makes this a fifteen-minute habit rather than a project; without it, it stops happening after the first month.
A launch checklist
- p50 and p95 response latency measured on the production voice and provider configuration, over a few hundred turns.
- Interruption cases pass, confirmed by listening, with correct conversation state afterwards.
- Eval set with transcript, audio and policy layers under version control; passing, with every failure explained.
- Simulated callers run through the real phone number at expected peak concurrency; p95 and error rate recorded.
- Concurrency, provider-failure, long-call and hang-up drills done.
- Twenty calls listened to by someone outside the project; notes acted on.
- Monitoring in place for the same latency components and a weekly listening quota on the calendar.
None of this is exotic. The difference between agents that launch well and ones that quietly get turned off is that the first group measured latency where the user feels it, broke their agent on purpose with interruptions, and kept listening after the launch.
Sources
- LiveKit Agents: Metrics and observability — https://docs.livekit.io/agents/build/metrics/
- LiveKit Agents: Session traces — https://docs.livekit.io/reference/agents/session-traces/
- Testing the Testers: Human-Driven Quality Assessment of Voice AI Testing Platforms (arXiv 2511.04133) — https://arxiv.org/abs/2511.04133
- Speechmatics: The 11 best voice agent testing platforms in 2026 — https://www.speechmatics.com/company/articles-and-news/de-risk-your-voice-agent-11-best-voice-agent-testing-platforms
- Cekura: How to test voice AI agents — https://www.cekura.ai/blogs/how-to-test-voice-ai-agents
- Ortavox pricing (concurrency, playground, tracing) — https://ortavox.ai/pricing