How to Make an AI Voice Assistant in Python (Step-by-Step)

Building an AI voice assistant in Python means chaining four things: a microphone stream with voice activity detection, a speech-to-text model, an LLM that writes the reply, and text-to-speech that speaks it. This guide builds all four in eight steps, with runnable code at every layer.

Every Python voice assistant tutorial produces the same demo. It answers one question beautifully, then goes deaf until you press a key again. This guide covers how to make an AI voice assistant in Python that listens like a person, with runnable code and the planning choices that come first.

TL;DR

• The record-for-five-seconds pattern copied across most Python tutorials is why your assistant cuts callers off mid-sentence and then sits in silence.
• Eight steps take you from python -m venv to a full conversation loop, with a local path and an API path at every layer.
• A loop that works on your laptop is a different thing from an assistant that works on a phone line, so simulate noise, accents, and barge-in before launch.

What You Need Before You Start Building in Python

Prerequisites:

• Python 3.11 or newer. The openai SDK requires 3.10 and above, while pipecat-ai requires 3.11, so starting at 3.11 keeps the later steps open. Grab it from python.org.
• PortAudio installed at the system level. The sounddevice package binds to it, and pip will error out without it.
• FFmpeg on your PATH for audio decoding and playback.
• An OpenAI API key for the reasoning step, or Ollama if you want the whole thing offline.
• About 500 MB of disk for the Whisper small.en weights, which download on first run.
• One defined task for the assistant. A clinic receptionist and an internal IT helper need different vocabularies and different silence thresholds.

Time required: 30 to 45 minutes for a working loop. Add half a day for streaming and interruption handling.

How to Make an AI Voice Assistant in Python: Step-by-Step

Eight steps, each one runnable on its own. Test every layer in isolation before wiring them together. Debugging a full loop with three possible culprits takes far longer than debugging three small scripts.

Step 1: Set Up the Project and Virtual Environment

Create a project folder, then an isolated environment so the audio and model packages stay off your system Python. Confirm the interpreter version first, since a 3.9 default surfaces as a cryptic wheel error four commands later.

mkdir voice-assistant && cd voice-assistant
python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
python -c "import sys; print(sys.version)"

Add a .env file for credentials and a .gitignore before you write a line of Python. Secrets committed on day one live in git history forever, and rotating a leaked key is a worse afternoon than creating a two-line file now.

printf 'OPENAI_API_KEY=sk-your-key-here\nELEVENLABS_API_KEY=your-key-here\n' > .env
printf '.venv/\n.env\n*.wav\n__pycache__/\n' > .gitignore

Pro tip: Run git init and commit the .gitignore as your first commit. Adding it after you have already committed .env does nothing to remove the key from history.

Step 2: Install the Audio and AI Libraries

This guide skips SpeechRecognition and pyttsx3, the stack most Python voice tutorials use. Both are blocking and offline-first, so neither supports the streaming and interruption handling that steps 3 and 8 depend on.

System audio libraries come first, then the Python packages. PortAudio and FFmpeg are C libraries that pip cannot supply, and installing them out of order produces build errors that look like Python problems and are not.

# macOS
brew install portaudio ffmpeg

# Debian or Ubuntu
sudo apt-get update && sudo apt-get install -y portaudio19-dev ffmpeg

Now pin the Python side. Pinned versions matter more here than in most projects, because the audio stack changes meaningfully between minor releases.

pip install \
  faster-whisper==1.2.1 \
  openai==3.3.1 \
  elevenlabs==2.64.0 \
  sounddevice==0.5.5 \
  webrtcvad-wheels==2.0.14 \
  numpy python-dotenv

A quick sanity check confirms your microphone is visible to Python before you build anything on top of it. Device index confusion accounts for a lot of silent recordings, especially on laptops with a webcam mic and a headset connected at once.

import sounddevice as sd

print(sd.query_devices())
print("Default input:", sd.default.device)

Pro tip: Install webrtcvad-wheels in place of the original webrtcvad package. It ships prebuilt wheels, which saves you from needing a C compiler on Windows.

Step 3: Capture Microphone Audio With Voice Activity Detection

This is the step that separates a usable assistant from a demo. Listen continuously and let voice activity detection decide when the caller has stopped talking. Fixed-length recording windows guess at something you can measure directly.

The reason is measurable. Stivers et al., studying ten languages in PNAS, found each language has a unimodal response-time distribution with a mode between 0 and +200 ms.

A hardcoded five-second window misses that target by an order of magnitude, which explains why tutorial assistants feel wrong to talk to.

Save this as listener.py:

import collections
import queue

import numpy as np
import sounddevice as sd
import webrtcvad

SAMPLE_RATE = 16000
FRAME_MS = 30
FRAME_SAMPLES = int(SAMPLE_RATE * FRAME_MS / 1000)   # 480 samples
FRAME_BYTES = FRAME_SAMPLES * 2                       # 16-bit mono


def record_utterance(aggressiveness=2, trailing_silence_ms=700, max_seconds=20):
    """Listen until the speaker stops, then return float32 audio."""
    vad = webrtcvad.Vad(aggressiveness)
    frames = queue.Queue()
    prebuffer = collections.deque(maxlen=int(300 / FRAME_MS))  # 300 ms lead-in
    collected = []
    speaking = False
    silent_frames = 0
    silence_limit = int(trailing_silence_ms / FRAME_MS)

    def callback(indata, frame_count, time_info, status):
        frames.put(bytes(indata))

    with sd.RawInputStream(samplerate=SAMPLE_RATE, blocksize=FRAME_SAMPLES,
                           dtype="int16", channels=1, callback=callback):
        for _ in range(int(max_seconds * 1000 / FRAME_MS)):
            frame = frames.get()
            if len(frame) != FRAME_BYTES:
                continue
            is_speech = vad.is_speech(frame, SAMPLE_RATE)
            if not speaking:
                prebuffer.append(frame)
                if is_speech:
                    speaking = True
                    collected.extend(prebuffer)
                    prebuffer.clear()
                continue
            collected.append(frame)
            silent_frames = 0 if is_speech else silent_frames + 1
            if silent_frames >= silence_limit:
                break

    if not collected:
        return np.zeros(0, dtype=np.float32)
    pcm = np.frombuffer(b"".join(collected), dtype=np.int16)
    return pcm.astype(np.float32) / 32768.0

The 300 ms prebuffer matters more than it looks. Voice activity detection always triggers slightly late, so without a lead-in you lose the first consonant of every sentence and your transcript reads "ello" in place of "hello."

Pro tip: WebRTC VAD accepts only 10, 20, or 30 ms frames at 8, 16, 32, or 48 kHz. Any other combination raises an error, so keep FRAME_MS and SAMPLE_RATE as a matched pair.

Step 4: Transcribe Speech with faster-whisper

faster-whisper reimplements Whisper on CTranslate2 and runs comfortably on CPU with int8 quantization. At roughly 25,000 GitHub stars, it is the practical default for local transcription in Python.

Model choice drives your latency budget. The small.en model hits the useful middle for English-only assistants, transcribing short utterances in roughly a second on a modern laptop CPU in our testing.

It stays far more accurate than tiny at that speed. Compare candidates on the Open ASR Leaderboard when you want numbers measured on your own audio.

A streaming speech-to-text provider fits better when you need partial transcripts during the utterance rather than after it, which is the route AssemblyAI and Deepgram document for live calls.

Save this as stt.py:

from faster_whisper import WhisperModel

# Loaded once at import time, never inside the loop.
_model = WhisperModel("small.en", device="cpu", compute_type="int8")


def transcribe(audio):
    if audio.size == 0:
        return ""
    segments, _info = _model.transcribe(
        audio,
        language="en",
        beam_size=1,          # greedy decoding, lowest latency
        vad_filter=False,     # our own VAD already trimmed the clip
        condition_on_previous_text=False,
    )
    return " ".join(segment.text for segment in segments).strip()

Two settings there earn their place. beam_size=1 trades a sliver of accuracy for a real speed gain. Disabling condition_on_previous_text stops one bad transcript from poisoning the next, a repetition loop that is painful to debug.

Pro tip: Run one throwaway transcription at startup to warm the model. The first call is always slower, and you would rather absorb that during boot than on a caller's opening question.

Step 5: Generate the Reply With an LLM

Voice output has constraints text chat does not. Your system prompt has to enforce short answers in plain words. Anything a screen renders silently gets read aloud literally, so a bulleted list becomes a caller hearing the word "asterisk."

Save this as brain.py

import os

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

SYSTEM_PROMPT = (
    "You are the phone receptionist for Northside Dental. "
    "Reply in one or two short sentences, always under 40 words. "
    "Write plain spoken words with no lists, markdown, or symbols. "
    "Say numbers as words. When something is unclear, ask one short question. "
    "For anything clinical, offer to transfer the caller to a nurse."
)

history = [{"role": "system", "content": SYSTEM_PROMPT}]
MAX_TURNS = 12


def generate_reply(user_text):
    history.append({"role": "user", "content": user_text})
    windowed = [history[0]] + history[-MAX_TURNS:] if len(history) > MAX_TURNS else history
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=windowed,
        max_tokens=120,
        temperature=0.4,
    )
    answer = response.choices[0].message.content.strip()
    history.append({"role": "assistant", "content": answer})
    return answer

Cost stays low at this size. GPT-4o mini runs $0.15 per million input tokens and $0.60 per million output tokens, so a short call costs a fraction of a cent. The MAX_TURNS window is what keeps it that way across a long conversation, capping both the bill and the growth in response time.

Going fully offline swaps three lines. Ollama exposes an OpenAI-compatible endpoint, so the client points at localhost, and the rest of the function is unchanged.

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
# then: model="llama3.2:3b"

Pro tip: Put the response length limit in the system prompt and in max_tokens. Prompts drift under pressure, and the hard cap is what protects you when the model decides to explain dental insurance in depth.

Step 6: Speak the Reply With Text-to-Speech

Two paths behind one function signature keep this swappable. Start with whichever matches your privacy constraints, then change your mind later without touching the loop. Save this as tts.py:

import os
import subprocess

from elevenlabs import ElevenLabs, play

_eleven = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])


def speak_api(text):
    audio = _eleven.text_to_speech.convert(
        text=text,
        voice_id="JBFqnCBsd6RMkjVDRZzb",
        model_id="eleven_flash_v2_5",     # lowest-latency tier
        output_format="mp3_22050_32",
    )
    play(audio)


def speak_local(text, model="en_US-lessac-medium.onnx"):
    subprocess.run(["piper", "--model", model, "--output_file", "reply.wav"],
                   input=text.encode(), check=True)
    subprocess.run(["ffplay", "-nodisp", "-autoexit", "-loglevel", "quiet",
                    "reply.wav"], check=True)


speak = speak_api    # swap to speak_local for a fully offline build

Piper runs on CPU and costs nothing per call, which suits high-volume or air-gapped builds. ElevenLabs sounds noticeably more natural, and its free tier gives you 10,000 credits monthly, which do not roll over to evaluate before committing to a paid plan.

Voice quality is worth auditioning on your own scripts before price enters the decision. Our TTS provider comparison covers how each provider holds up on short functional replies like confirmations and handoffs.

Pro tip: Pick the fastest model tier your quality bar allows. Time to first audio matters far more to a caller than the polish of a voice they hear two seconds late.

Step 7: Assemble the Conversation Loop

Every piece now exists as its own function, so the loop itself is twenty lines. Listen, transcribe, think, speak, repeat.

from listener import record_utterance
from stt import transcribe
from brain import generate_reply
from tts import speak


def main():
    print("Listening. Ctrl+C to quit.")
    speak("Thanks for calling Northside Dental. How can I help?")
    while True:
        audio = record_utterance()
        caller_text = transcribe(audio)
        if not caller_text:
            continue
        print(f"Caller:    {caller_text}")
        answer = generate_reply(caller_text)
        print(f"Assistant: {answer}")
        speak(answer)


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        print("\nSession ended.")

Run it with python main.py and talk. A cough or a closing door triggers voice activity detection and produces a blank transcript. The if not caller_text check catches that and waits for the next utterance, so the model never sees an empty turn.

Pro tip: Log the elapsed milliseconds for each stage now, while the code is small. You will want that breakdown in the next step, and adding timers retroactively across four modules is tedious.

Step 8: Stream the Pipeline to Cut Response Time

The loop above waits for the full reply before speaking a single word. Streaming removes that dead air by starting synthesis on the first finished sentence while the model is still writing the second.

import re
import queue
import threading

SENTENCE_END = re.compile(r"(?<=[.!?])\s+")


def _speaker_worker(sentences):
    while True:
        sentence = sentences.get()
        if sentence is None:
            return
        speak(sentence)


def stream_reply(user_text):
    history.append({"role": "user", "content": user_text})
    sentences = queue.Queue()
    worker = threading.Thread(target=_speaker_worker, args=(sentences,), daemon=True)
    worker.start()

    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[history[0]] + history[-MAX_TURNS:],
        max_tokens=120,
        stream=True,
    )

    buffer, full_reply = "", ""
    for chunk in stream:
        delta = chunk.choices[0].delta.content or ""
        if not delta:
            continue
        buffer += delta
        full_reply += delta
        while (match := SENTENCE_END.search(buffer)):
            sentence, buffer = buffer[:match.start()], buffer[match.end():]
            sentences.put(sentence.strip())

    if buffer.strip():
        sentences.put(buffer.strip())
    sentences.put(None)
    worker.join()
    history.append({"role": "assistant", "content": full_reply.strip()})
    return full_reply.strip()

Measure time to first audio in place of total response time. Per Cekura's benchmarks, mean response time across seven production voice platforms ranges from 1.27 seconds on ElevenLabs to 3.08 seconds on Vapi, measured over 82 scenarios run three times each.

A caller judges the pause before you speak, and this change removes the synthesis wait on every sentence after the first, so the caller hears audio as soon as sentence one is complete rather than after the full reply

Pro tip: Keep the sentence queue bounded with queue.Queue(maxsize=3). If the caller interrupts, a bounded queue is far easier to drain than an unbounded one holding six pending sentences.

Common Mistakes to Avoid

• Recording for a fixed number of seconds: The most copied pattern in Python voice tutorials, and the one that cuts callers off mid-word. Step 3 replaces it with voice activity detection, and our guide to endpointing and turn detection covers the tuning in depth.
• Loading the speech model inside the loop: A WhisperModel(...) call per turn adds seconds every single time. Load once at import, warm it at startup.
• Letting conversation history grow without limit: Token cost and time to first token both increase with each turn. Window the history or summarize older turns.
• Blocking the audio thread during playback: Synchronous playback makes the assistant deaf while it talks, so an interruption lands nowhere. Push speech to a worker thread, as in step 8.
• Hardcoding API keys in source: Committed keys are scraped from public repositories, which is why GitHub runs secret scanning and providers auto-revoke on detection.
• Testing only on your own clean audio: Your desk mic, your accent, and your phrasing represent almost none of your real callers. Word error rates on the Open ASR Leaderboard are measured on clean read speech, so treat them as a ceiling rather than what you will see on noisy phone audio.

Taking It Further: Interruptions, Tool Calls, and Telephony

Four upgrades separate a working prototype from something you would put on a phone number.

Barge-in means the assistant stops talking the moment a caller starts. Keep voice activity detection running during playback, cancel the audio stream on detected speech, and record only the spoken portion in history.

Skipping that last part causes a specific and confusing bug, where the assistant believes it said something the caller never heard.

Wake word detection lets the assistant sit idle until it hears its name. A keyword-spotting engine such as Porcupine runs on the raw audio stream and wakes the pipeline only on a match, so ambient conversation never reaches transcription or the model. Check the licensing before you ship.

Porcupine's built-in keywords ship under Apache 2.0, while custom wake words trained on the free Personal tier carry a non-commercial restriction.

Tool calling turns an answering machine into something useful. The model returns a structured function request, your code executes it against a calendar or CRM, and the result feeds back as another message.

Confirm before anything irreversible, since one extra spoken turn costs less than a wrongly canceled appointment.

Telephony is where hand-rolled pipelines start to strain. Frameworks like Pipecat, a BSD-2 licensed project maintained by Daily, handle transport, turn-taking, and interruption as solved problems.

Our walkthrough on building LiveKit agents covers the other common route. The conversational voice AI primer explains the timing constraints both frameworks address.

How to Test Your Python Voice Assistant Before Real Callers Arrive

Your assistant works on your laptop, in a quiet room, with you asking the questions you designed it to answer. But that combination hides almost every problem that matters.

Real callers arrive with accents your speech recognition has trouble with, background noise that confuses voice activity detection, and phrasing no test script anticipated. They talk over the assistant, change their mind mid-sentence, and answer a question with a question.

Manual testing cannot reach any of that at useful volume. Calling your own agent two hundred times can only teach you so much.

Four categories are worth structured coverage before launch:

• Transcription accuracy on noisy audio carrying your domain vocabulary.
• Interruption recovery when a caller barges in during a long reply.
• Off-script handling when someone asks about something outside scope.
• Regression coverage after every prompt edit, because a one-word change can undo a working flow without visible warning.

The full testing checklist walks through each of these in sequence.

Cekura Makes Voice Assistant Testing Easier

Knowing how to make an AI voice assistant in Python gets you a working pipeline, but proving it holds up under real conditions is separate work, and Cekura runs that layer on top of whatever you built.

Pre-production:

• Simulation at scale: Cekura runs thousands of multi-turn conversations against your agent, driven by personas carrying different accents, speaking speeds, and interruption habits.
• Red teaming: Adversarial runs covering jailbreak attempts, prompt injection, and data extraction, so a caller cannot talk your assistant out of its own rules.

Infrastructure:

• Infrastructure checks: Background noise, latency spikes, poor audio quality, and barge-in timing, all measured by Cekura against the raw audio signal.
• Load testing: Concurrency runs that surface the point where response times degrade, before a campaign finds it for you.

Observability:

• Production scoring: Every live call evaluated automatically on latency, interruption handling, and instruction following, with alerts when quality drifts. Our seven-method guide covers how these layers stack.

Already on a managed stack? Cekura offers native integrations for Retell, VAPI, ElevenLabs, LiveKit, Pipecat, Bland, and more.

Custom Python pipelines like the one you just built connect to Cekura through a webhook integration. Nothing in your code needs rewriting.

Handling patient data or payment details? Cekura supports SOC 2, HIPAA, and GDPR compliance, covering transcript redaction, role-based access, and audit trails.

Want to see your own agent stress-tested? Book a demo to watch Cekura run simulated callers, noise conditions, and interruption scenarios against your Python voice assistant.

Frequently Asked Questions

What Python version do I need to build an AI voice assistant?

Python 3.11 or newer covers every library in this guide. The openai SDK requires 3.10 and above, and pipecat-ai requires 3.11, so starting at 3.11 avoids reinstalling your environment when you add a framework later.

Can I make an AI voice assistant in Python without any API keys?

Yes, you can run the entire pipeline offline with faster-whisper for transcription, Ollama for the language model, and Piper for speech output. There is no per-call cost. Expect higher latency on CPU and a less natural voice than a paid provider gives you.

Which Python library is best for speech recognition?

faster-whisper is the strongest default for local speech recognition in Python. It runs on CPU with int8 quantization and transcribes short utterances in well under a second. A streaming provider suits you better when you need results word by word during a live call.

Do I need a GPU to run a Python voice assistant?

No, a GPU is optional for a single-user assistant. The small.en Whisper model and a 3B-parameter local language model both run on a modern CPU. A GPU helps once you move to larger models or handle several concurrent calls.

How do I reduce the response time of my voice assistant?

Start synthesis on the first completed sentence while the model is still writing, keep responses to one or two sentences, and cap conversation history. Measuring time to first audio in place of total response time also points you at the stage actually costing the delay.

Is a Python voice assistant the same thing as a voice agent?

The main difference between a voice assistant and a voice agent is action. An assistant answers questions, while an agent completes tasks in connected systems such as booking an appointment or updating a record. Our guide to planning an assistant build covers which one your use case calls for.