All blog posts
EngineeringLatencyVoice AIArchitecture

How to Achieve Sub-600ms AI Voice Latency: A Technical Deep Dive

March 15, 2025 · 3 min read · Ortavox Engineering

A sequential voice pipeline easily exceeds 2,000ms per turn. Here is exactly how Ortavox achieves sub-600ms median end-to-end latency through parallel pipeline processing, edge VAD, streaming LLM inference, and overlapped TTS synthesis.

Human conversation has a natural rhythm: the gap between when one person stops speaking and the other starts is typically a few hundred milliseconds. When an AI voice agent takes much longer than that to respond, the conversation feels robotic. When it takes 2,000ms or more — which is what a naive sequential pipeline produces — it actively damages user trust.

At Ortavox, we obsess over this number. Our representative p50 end-to-end latency is ~540ms, measured from the moment the user stops speaking to the moment the first audio byte reaches the caller. Here is exactly how we achieve it.

Why most platforms are slow

The typical voice AI pipeline is sequential: wait for speech to end → send to STT → get full transcript → send to LLM → wait for complete response → send to TTS → wait for audio → stream to caller. This waterfall easily exceeds 2,000ms even with fast providers.

Stage

Sequential (illustrative)

Ortavox (parallel, p50)

VAD end-of-speech

500–1,500ms (silence timeout)

~20ms (probabilistic)

STT transcription

Wait for the final transcript

~80ms (streaming)

LLM first token

Wait for the full response

~220ms

TTS first audio

Wait for the full text

~180ms (streaming)

Total

Easily 2,000ms+

~540ms (stages overlap, so the total is not the sum)

The four optimizations that matter

1. Probabilistic VAD instead of silence timeouts

Traditional VAD waits for 500–1,500ms of silence before declaring end-of-speech. Ortavox runs a lightweight VAD model directly in the WebSocket connection layer, on every 20ms audio frame, producing a probabilistic end-of-utterance score. When the score crosses a threshold, transcription is triggered within about 50ms of actual speech end — no silence timeout. This removes the single largest source of dead time from every turn.

2. Streaming STT with immediate LLM forwarding

Rather than waiting for a final transcript, Ortavox streams audio to the STT provider and forwards interim transcripts to the LLM as they arrive. Deepgram Nova-3 is the default because it returns interim transcripts in real time; Whisper, Google, Groq, Cartesia Ink, Gladia, Speechmatics and Mistral Voxtral are also supported. By the time the final transcript lands, the LLM has already processed most of the input and is generating tokens.

3. Sentence-boundary TTS triggering

Ortavox monitors the LLM token stream for natural sentence boundaries. The moment a complete sentence is available, it is forwarded to TTS for synthesis. The first audio chunk reaches the caller before the LLM has finished generating the full response, which is why TTS first-audio and LLM generation overlap rather than add up.

4. WebSocket persistence and audio buffer management

Per-request HTTP connections add handshake overhead to every turn. Ortavox maintains persistent WebSocket connections for the entire call. Audio chunks from TTS are queued in a small buffer (1–2 sentences ahead) and streamed continuously.

Representative latency breakdown

The following is a representative breakdown for a typical turn, measured from end-of-user-speech to first audio byte delivered to the caller. Stages run in parallel, so the total is not the sum of the individual stages.

Stage

p50

p95

VAD end-of-speech detection

~20ms

~50ms

STT transcription (streaming)

~80ms

~150ms

LLM first token (TTFT)

~220ms

~380ms

TTS first audio byte

~180ms

~250ms

Network + buffer overhead

~40ms

~80ms

Total (first audio to caller)

~540ms

~910ms

Sources

  • Ortavox — How it works (pipeline stages, supported providers and latency breakdown) — https://ortavox.ai/how-it-works

Ready to build?

Start with 100 free minutes. No credit card required.

Get started free