voice-agents

Voice Agents

Voice agents represent the frontier of AI interaction - humans speaking

naturally with AI systems. The challenge isn't just speech recognition

and synthesis, it's achieving natural conversation flow with sub-800ms

latency while handling interruptions, background noise, and emotional

nuance.

This skill covers two architectures: speech-to-speech (OpenAI Realtime API,

lowest latency, most natural) and pipeline (STT→LLM→TTS, more control,

easier to debug). Key insight: latency is the constraint. Humans expect

responses in 500ms. Every millisecond matters.

84% of organizations are increasing voice AI budgets in 2025. This is the

year voice agents go mainstream.

Detailed Guide

Read [the detailed guide](references/detailed-guide.md) before executing this skill. It retains the complete procedure and reference material. Treat its safety, prerequisites, and validation requirements as mandatory. For focused work, load the relevant sections; for end-to-end work, read the guide completely.

Production Pipeline Example

"""

import { Deepgram } from '@deepgram/sdk';

import { ElevenLabsClient } from 'elevenlabs';

import OpenAI from 'openai';

// Initialize clients

const deepgram = new Deepgram(process.env.DEEPGRAM_API_KEY);

const elevenlabs = new ElevenLabsClient();

const openai = new OpenAI();

async function processVoiceInput(audioStream) {

// 1. Speech-to-Text (Deepgram Nova-3)

const transcription = await deepgram.transcription.live({

model: 'nova-3',

punctuate: true,

endpointing: 300, // ms of silence before end

});

transcription.on('transcript', async (data) => {

if (data.is_final && data.speech_final) {

const userText = data.channel.alternatives[0].transcript;

console.log('User:', userText);

// 2. LLM Processing

const completion = await openai.chat.completions.create({

model: 'gpt-4o-mini',

messages: [

{ role: 'system', content: 'You are a concise voice assistant.' },

{ role: 'user', content: userText }

],

max_tokens: 150, // Keep responses short for voice

});

const agentText = completion.choices[0].message.content;

console.log('Agent:', agentText);

// 3. Text-to-Speech (ElevenLabs)

const audioStream = await elevenlabs.textToSpeech.stream({

voice_id: 'voice_id_here',

text: agentText,

model_id: 'eleven_flash_v2_5', // Lowest latency

});

// Stream to user

playAudioStream(audioStream);

}

});

// Pipe audio to transcription

audioStream.pipe(transcription);

}

"""

Optimization Tips:

• Start TTS while LLM still generating (streaming)
• Pre-compute first response segment during user speech
• Use Flash/turbo models for latency

Voice Activity Detection Pattern

Detect when user starts/stops speaking

When to use: All voice agents need VAD for turn-taking

VOICE ACTIVITY DETECTION (VAD):

"""

VAD Types:

1. Energy-based: Simple, fast, noise-sensitive
2. Model-based: Silero VAD, more accurate
3. Semantic VAD: Understands meaning, best for conversation

"""

When to Use

• User mentions or implies: voice agent
• User mentions or implies: speech to text
• User mentions or implies: text to speech
• User mentions or implies: whisper
• User mentions or implies: elevenlabs
• User mentions or implies: deepgram
• User mentions or implies: realtime api
• User mentions or implies: voice assistant
• User mentions or implies: voice ai
• User mentions or implies: conversational ai
• User mentions or implies: tts
• User mentions or implies: stt
• User mentions or implies: asr

Limitations

• Use this skill only when the task clearly matches the scope described above.
• Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
• Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.