How to Achieve Sub-600ms AI Voice Latency: A Technical Deep Dive
March 15, 2025 · 3 min read · Ortavox Engineering
A sequential voice pipeline easily exceeds 2,000ms per turn. Here is exactly how Ortavox achieves sub-600ms median end-to-end latency through parallel pipeline processing, edge VAD, streaming LLM inference, and overlapped TTS synthesis.
Human conversation has a natural rhythm: the gap between when one person stops speaking and the other starts is typically a few hundred milliseconds. When an AI voice agent takes much longer than that to respond, the conversation feels robotic. When it takes 2,000ms or more — which is what a naive sequential pipeline produces — it actively damages user trust.
At Ortavox, we obsess over this number. Our representative p50 end-to-end latency is ~540ms, measured from the moment the user stops speaking to the moment the first audio byte reaches the caller. Here is exactly how we achieve it.
Why most platforms are slow
The typical voice AI pipeline is sequential: wait for speech to end → send to STT → get full transcript → send to LLM → wait for complete response → send to TTS → wait for audio → stream to caller. This waterfall easily exceeds 2,000ms even with fast providers.
Stage | Sequential (illustrative) | Ortavox (parallel, p50) |
|---|---|---|
VAD end-of-speech | 500–1,500ms (silence timeout) | ~20ms (probabilistic) |
STT transcription | Wait for the final transcript | ~80ms (streaming) |
LLM first token | Wait for the full response | ~220ms |
TTS first audio | Wait for the full text | ~180ms (streaming) |
Total | Easily 2,000ms+ | ~540ms (stages overlap, so the total is not the sum) |
The four optimizations that matter
1. Probabilistic VAD instead of silence timeouts
Traditional VAD waits for 500–1,500ms of silence before declaring end-of-speech. Ortavox runs a lightweight VAD model directly in the WebSocket connection layer, on every 20ms audio frame, producing a probabilistic end-of-utterance score. When the score crosses a threshold, transcription is triggered within about 50ms of actual speech end — no silence timeout. This removes the single largest source of dead time from every turn.
2. Streaming STT with immediate LLM forwarding
Rather than waiting for a final transcript, Ortavox streams audio to the STT provider and forwards interim transcripts to the LLM as they arrive. Deepgram Nova-3 is the default because it returns interim transcripts in real time; Whisper, Google, Groq, Cartesia Ink, Gladia, Speechmatics and Mistral Voxtral are also supported. By the time the final transcript lands, the LLM has already processed most of the input and is generating tokens.
3. Sentence-boundary TTS triggering
Ortavox monitors the LLM token stream for natural sentence boundaries. The moment a complete sentence is available, it is forwarded to TTS for synthesis. The first audio chunk reaches the caller before the LLM has finished generating the full response, which is why TTS first-audio and LLM generation overlap rather than add up.
4. WebSocket persistence and audio buffer management
Per-request HTTP connections add handshake overhead to every turn. Ortavox maintains persistent WebSocket connections for the entire call. Audio chunks from TTS are queued in a small buffer (1–2 sentences ahead) and streamed continuously.
Representative latency breakdown
The following is a representative breakdown for a typical turn, measured from end-of-user-speech to first audio byte delivered to the caller. Stages run in parallel, so the total is not the sum of the individual stages.
Stage | p50 | p95 |
|---|---|---|
VAD end-of-speech detection | ~20ms | ~50ms |
STT transcription (streaming) | ~80ms | ~150ms |
LLM first token (TTFT) | ~220ms | ~380ms |
TTS first audio byte | ~180ms | ~250ms |
Network + buffer overhead | ~40ms | ~80ms |
Total (first audio to caller) | ~540ms | ~910ms |
Sources
- Ortavox — How it works (pipeline stages, supported providers and latency breakdown) — https://ortavox.ai/how-it-works