How to Build a Voice Agent

A voice agent enables real-time spoken conversation with an AI. It combines speech-to-text (STT), an LLM for reasoning, and text-to-speech (TTS) into a pipeline that processes audio streams with sub-second latency. This guide covers the architecture, component selection, and production deployment.

The standard voice agent pipeline has three stages:

Audio In → **STT** → Text → **LLM** → Text → **TTS** → Audio Out

Each stage runs as a streaming component, processing data incrementally rather than waiting for complete inputs:

• STT receives audio chunks (100-250ms segments) and emits partial transcripts
• LLM receives the transcript and streams token-by-token responses
• TTS converts text chunks to audio as they arrive, without waiting for the full response

The streaming pattern is the production standard. An orchestration layer manages the data flow between components, handles interruptions, and maintains conversation state. ¹⁾

Bidirectional data streaming over HTTP. The client sends audio chunks and receives audio/text responses on the same connection. Ideal for web and cloud-based applications.

Peer-to-peer audio/video with built-in NAT traversal, echo cancellation, and noise suppression. Better for telephony integration (SIP trunking) and scenarios requiring the lowest possible latency.

Both protocols support the streaming pipeline. WebSocket is simpler to implement; WebRTC provides better audio quality and telephony compatibility. ²⁾

Deepgram leads for production voice agents due to its streaming latency and accuracy. Whisper is best for self-hosted or offline scenarios where latency is less critical. ³⁾

Cartesia excels when latency is the top priority. ElevenLabs produces the most human-like voices for agents where personality and expression matter. ⁴⁾

Platforms provide pre-built orchestration so you do not need to wire together STT, LLM, and TTS manually:

Vapi and LiveKit are for teams that want control over each pipeline component. Retell and Bland are for shipping fast with less engineering effort. ⁵⁾

The LLM sits at the center of the pipeline, receiving transcripts and generating responses:

• Conversation history – maintain a message array with previous turns for context
• Function calling – enable the LLM to trigger actions (book appointments, query databases)
• RAG integration – retrieve knowledge base documents to ground responses
• Response formatting – instruct the LLM to respond in short, spoken-friendly sentences
• Sentiment awareness – adjust tone and content based on detected user emotion

Optimize LLM prompts for spoken output: shorter sentences, no markdown, no code blocks, conversational tone. ⁶⁾

Target: under 1 second end-to-end (STT ~300ms + LLM ~300ms + TTS ~200ms).

• Stream everything – do not wait for complete outputs at any stage
• Use fast STT/TTS providers – Deepgram for STT, Cartesia for TTS minimizes pipeline latency
• Choose a fast LLM – smaller models (GPT-4o-mini, Llama 3 8B) reduce time-to-first-token
• Edge deployment – run components geographically close to users
• VAD (Voice Activity Detection) – detect speech endpoints quickly to minimize silence before processing
• Prefetch and cache – preload common responses, cache TTS for frequent phrases

Measure Time-to-First-Token (TTFT) and Time-to-First-Audio as primary latency metrics. ⁷⁾

Users frequently interrupt (barge-in). The agent must:

1. Detect user speech onset via VAD while the agent is speaking
2. Immediately stop TTS playback
3. Flush the current LLM generation
4. Process the new user input from the interruption point
5. Resume the conversation naturally (e.g., Sorry, go ahead)

Platforms like LiveKit and Vapi handle interruption natively. For custom builds, implement a state machine that transitions between listening, thinking, speaking, and interrupted states. ⁸⁾

Follow a phased approach:

1. Define KPIs – containment rate (40-60% target), average handle time, user satisfaction (CSAT)
2. Prepare integrations – connect to telephony (Twilio), CRM, knowledge base
3. Alpha test – internal red-teaming with adversarial scenarios
4. Beta test – route 10% of traffic to the voice agent, A/B test against human agents
5. Scale – expand traffic gradually, monitor KPIs, iterate

Voice agent costs are typically $0.05-0.20 per minute depending on the provider stack. Major cost components:

• STT: $0.005-0.02/minute
• LLM: $0.01-0.05/minute (depends on model and verbosity)
• TTS: $0.01-0.05/minute
• Telephony: $0.01-0.03/minute

⁹⁾

• How to Build an AI Assistant
• How to Use Function Calling
• How to Self-Host an LLM