All blog posts
EngineeringArchitectureBYOKVoice AI

Ortavox Engine, Explained: The Voice Pipeline, BYOK and What You Actually Configure

September 18, 2026 · 7 min read · Ortavox Engineering

A developer's tour of the Ortavox voice engine: how STT, LLM, TTS, turn-taking and telephony fit together on LiveKit, how bring-your-own-key works without your keys ever touching the media path, and what an agent configuration really contains.

Most voice-agent platforms describe themselves with a diagram of four boxes: speech-to-text, a language model, text-to-speech, and a phone. That diagram is not wrong, but it hides the parts that decide whether a call feels good: who detects the end of a turn, what happens when the caller interrupts, where the API keys live, and how a call gets from an HTTP request to a running worker. This article walks through the Ortavox engine as it is built, for developers deciding whether to run on it.

Three components, one media server

Ortavox is organised as three deployable units. The API (TypeScript, Express) owns agents, credentials, calls, telephony configuration and billing. The core (Python) is the conversation engine: it runs the per-call worker that does STT, LLM, TTS, tools, memory and interruption handling. The UI is the console and the browser playground. All real-time media goes through LiveKit, which carries WebRTC for browser calls and SIP for phone calls.

The call path is deliberately boring. The API creates a room and a token, writes the call to Postgres, and publishes a dispatch command on a Redis stream. A dispatcher service checks the account's concurrency limit, queues the call if the limit is hit, and otherwise asks LiveKit to dispatch an agent into the room. A core worker, registered with LiveKit at startup, picks up the job, rebuilds the agent configuration from the dispatch metadata and joins the room. Nothing in that path is bespoke infrastructure; it is Redis streams, Postgres and LiveKit's agent dispatch.

The pipeline inside a call

Inside the worker the engine is a cascaded pipeline built on the LiveKit Agents SDK. Each stage is a pluggable provider resolved from a registry, so the same worker binary serves an agent that uses Deepgram, GPT and Cartesia and one that uses Speechmatics, Claude and ElevenLabs.

Stage

What it does

Providers in the registry today

VAD

Detects speech vs silence on the inbound audio frames

Silero (default), Silero combined with Krisp on LiveKit Cloud

Noise cancellation

Cleans the caller's audio before STT

Krisp (LiveKit Cloud deployments), RNNoise (self-hosted deployments)

STT

Streams partial and final transcripts

Deepgram, OpenAI, Google, Groq, Mistral (Voxtral), Speechmatics, Azure, AssemblyAI, Cartesia

Turn detection

Decides when the user has finished speaking

English model, multilingual model, manual (push-to-talk style), realtime (delegated to a speech-to-speech model)

LLM

Produces the reply and tool calls, streamed token by token

OpenAI, Anthropic, Google, Groq, Mistral, Azure, AWS Bedrock

TTS

Synthesises the streamed reply

Cartesia, ElevenLabs, OpenAI, Google, Deepgram, Groq, Mistral, Azure, Rime, Neuphonic, Kyutai (self-hosted), Ultravox

Two things in that table matter more than the brand names. First, turn detection is a separate stage from VAD. VAD only tells you that audio stopped; the turn-detection model looks at the transcript so far and estimates whether the person is done or just pausing mid-sentence. That is what lets the engine answer quickly after "yes" and wait after "so what I wanted to ask was". Second, noise cancellation is chosen per deployment, not per agent: on LiveKit Cloud the engine uses Krisp's room-level models, and on self-hosted LiveKit it runs RNNoise inside the STT path on the agent server. The agent configuration does not change between the two.

Telephony

Phone calls enter and leave through LiveKit SIP. The API holds SIP trunk configuration and creates outbound SIP participants when an agent places a call; inbound calls arrive as SIP participants in a room, and the same dispatch flow puts a worker next to them. You can use Twilio, Telnyx or your own SIP trunk; Ortavox does not resell minutes, so carrier charges are billed by the carrier. Browser calls use WebRTC and no carrier at all, which is why the pricing page tells you to set the carrier rate to zero for them.

The engine also runs against two LiveKit backends: a self-hosted LiveKit cluster and LiveKit Cloud. Workers connect to exactly one backend each, so cloud and self-hosted capacity are separate worker pools with separate SIP trunks. Which backend an account lands on is an operator decision, not something the agent configuration needs to know about.

What an agent configuration contains

When the API dispatches a call it serialises the agent into metadata that the worker reconstructs. Stripped to the parts a developer sets, it looks like this:

json
{
  "stt": { "provider": "deepgram", "model": "nova-3", "language": "en" },
  "llm": { "provider": "openai", "model": "gpt-4.1-mini", "temperature": 0.3,
           "credential_id": "cred_7f3a..." },
  "tts": { "provider": "cartesia", "model": "sonic-3", "voice": "..." },
  "turn_detection": { "provider": "multilingual" },
  "system_prompt": "You are the booking assistant for ...",
  "start_message": "Hi, this is the clinic. How can I help?",
  "tools": [ { "name": "check_availability", "parameters": { } } ]
}

Every block carries a provider and a model, plus provider-specific extras that are passed straight through to the underlying plugin. The one field worth a closer look is credential_id.

How BYOK works

Bring-your-own-key on Ortavox is designed around one rule: customer keys never travel through the media layer. Dispatch metadata passes through the LiveKit server, and on LiveKit Cloud that server is a third party. So the metadata only ever contains a credential ID.

  1. You add a key with POST /v1/credentials. The API makes one cheap read-only call to the provider to validate it, encrypts it with AES-256-GCM and stores the ciphertext in Postgres with a last-four-characters hint for the UI. The raw key exists in memory only inside that request.
  2. You attach the credential to an agent's STT, LLM or TTS block. An agent can mix: your own OpenAI key for the LLM, Ortavox-managed keys for everything else.
  3. At call start the worker sees credential_id in the block, fetches the decrypted key from the API over the internal server-to-server channel, and passes it to the provider plugin as api_key.
  4. If a block has no credential, the plugin falls back to the Ortavox-managed key from the worker's environment. There is no mode flag; a null credential is the managed path.

Deleting a credential nulls the reference on any agent using it, so the agent silently returns to managed keys rather than failing. On the billing side, a call that ran on your key is not metered for that provider's cost; you pay the platform rate and the provider bills you directly. The pricing page is explicit that BYOK does not make model usage free, it just moves the invoice.

✓

The validation call at credential creation is the cheapest place to catch a bad key. A key that fails there never reaches a live call; a key that expires later fails the call and the log says which provider rejected it.

Provider routing and managed spend

For managed keys, the engine tracks raw provider cost per call and whether each stage ran on a customer key or an Ortavox key. That feeds an internal ledger of provider balances so the team knows which upstream account needs a top-up before calls start failing. It is invisible to customers, but it is why a managed account on Ortavox does not degrade when one provider's prepaid balance runs low.

Routing itself is static per agent. There is no automatic failover from one STT vendor to another mid-call, and we do not advertise one. In practice a cross-provider failover changes the voice, the transcript style and the latency profile of the call, and most customers would rather choose the fallback explicitly than discover it on a recording.

Latency: what the published numbers mean

The how-it-works page publishes a stage budget: roughly 20 to 50 ms for end-of-speech detection, 80 to 150 ms for streaming STT, 220 to 380 ms for LLM first token, 180 to 250 ms for TTS first audio and 40 to 80 ms of network and buffering, for a total of about 540 ms at the median and 910 ms at p95 with a fast model stack. Treat those as design targets for a well-chosen configuration, not a guarantee for any configuration. A large reasoning model as the LLM, or a high-quality TTS tier, will move the total by hundreds of milliseconds, and the console's turn-by-turn latency waterfall exists so you can see which stage did it.

Deployment options

Pay-as-you-go runs on Ortavox's cloud at $0.055 per connected minute plus providers at cost, with 100 free test minutes and 4 concurrent calls included. Above roughly 50K minutes a month rates step down toward $0.040. Enterprise deployments can run the same three components on your own Kubernetes or Docker infrastructure, on your GPUs, with no audio leaving your network; the engine's choice of RNNoise and self-hosted LiveKit is what makes that possible without a dependency on LiveKit Cloud's licensed audio models.

Sources

  • Ortavox core, API and deployment documentation (internal)
  • Ortavox pricing and how-it-works pages — https://ortavox.ai/pricing, https://ortavox.ai/how-it-works
  • LiveKit Agents documentation — https://docs.livekit.io/agents/

Ready to build?

Start with 100 free minutes. No credit card required.

Get started free