To test voice AI agents from your IDE, run text-mode checks locally with LiveKit’s agent debugger or Pipecat Evals, have your coding assistant run simulated calls through Cekura’s MCP server or CLI, and gate the merge in CI. Vapi, Retell and ElevenLabs agents start with Cekura. Cekura scores each call and returns pass or fail to the editor.
TL;DR
- Testing a voice agent from your IDE has five layers: local framework tests, Cekura’s MCP server, Skills, CLI and Python SDK, and a CI gate.
- LiveKit’s
lk agent debuggerruns in text mode by default, so a turn costs only your agent’s LLM and tool calls, but text mode cannot catch digits transcribed as words. - Cekura’s MCP server exposes one
scenarios_run_*tool per connection type, including telephony, SIP, Vapi, Retell and ElevenLabs, so the assistant must match the tool to the agent’s configured connection. - A Cekura result marked
completedhas not necessarily passed, andcekura run startwith--wait --fail-on-threshold 1.0exits 1 unless every run passed. - In Cekura’s voice agent workflow benchmark, the configuration that ranked first on repeatable reliability (Retell) scored 75.6% pass³, the share of 82 scenarios where all three runs passed every applicable configured rubric gate, and 93.9% task completion per call; providers chose their own stacks.
What does it mean to test a voice AI agent from your IDE?
Testing a voice AI agent from your IDE is a workflow in which you, or the coding assistant in your editor, start test conversations, read the transcripts and scores, and fix the failure without opening a dashboard. The call runs against the agent under test, and the IDE is only the control surface.
The workflow has five layers:
- Framework-native local tests run the agent on your machine: LiveKit’s CLI agent debugger, unit tests and simulations, or Pipecat Evals.
- The Cekura MCP server lets Cursor, VS Code or Claude Code list agents, run scenarios and read results as tool calls.
- Cekura Skills teach the assistant how to design evaluators, pick metrics and triage a failed run.
- The Cekura CLI and Python SDK run the same tests from the IDE terminal or a script.
- A CI gate reruns the suite on every pull request.
Cekura covers the last four layers and connects to LiveKit and Pipecat agents directly, so local framework tests stay in place.
Which method should you use to test voice AI agents from your IDE?
The right method for testing voice AI agents from your IDE is the cheapest layer that can reach the failure you are chasing. A prompt that calls the wrong tool shows up in a text test. A caller’s account number misheard over a phone line only shows up on the speech path.
| Method | Who drives it | Reaches real audio and transport | Cost basis | Use it for |
|---|---|---|---|---|
| LiveKit agent debugger and unit tests (layer 1) | The assistant via lk agent debugger, or pytest and Vitest | Text by default; --audio runs the agent’s full STT, turn detection and TTS pipeline, but each turn waits for the agent to finish, so it cannot test barge-in | Your agent’s LLM and tool calls; audio mode adds LiveKit Inference | Checking every edit before calling it done |
| Pipecat Evals (layer 1) | pipecat eval run, from you or the assistant | Text by default; audio mode synthesizes the caller, but the local eval transport is not a real WebSocket, SIP or telephony path | Text mode skips STT and TTS usage | Scripted turns, tool-call assertions, latency budgets |
| Cekura MCP server (layer 2) | The assistant, as tool calls in chat | Yes: telephony, SIP, provider WebRTC, LiveKit, Pipecat, or text | $0.25 per voice testing minute, $0.025 per text reply | Running and triaging simulated calls without leaving the editor |
| Cekura Skills (layer 3) | The assistant’s workflow guidance | Only when paired with MCP | Included on every plan; usage starts when a run starts | Planning coverage and reading failures |
| Cekura CLI and SDK (layer 4) | You, a script, or the assistant’s shell | Yes, set by --mode | Same usage rates | Scripted runs, committed test specs, exit codes |
| CI gate (layer 5) | GitHub Actions on each pull request | Yes, set by execution_mode | Same usage rates | Blocking a merge that breaks a scenario |
Cekura’s docs split Skills and MCP this way: with Skills alone the assistant can advise, with MCP alone it can call tools, and with both it can pick the right workflow and finish it. The local layer covers LiveKit and Pipecat. If your agent runs on Vapi, Retell or ElevenLabs, start with Cekura’s MCP server, which has a run tool for each of those connections. Pair one local layer with one Cekura layer while you code, then add the CI gate. You trade money for fidelity: a local text turn costs only your agent’s LLM and tool calls, while a Cekura voice run bills per minute but exercises the path a caller hears.
Why is a text-mode test in your IDE not enough to test a voice AI agent?
A text-mode test is a check that sends the caller’s turns as text and skips speech recognition and synthesis, so it proves logic and tool calls, not what a caller hears. LiveKit’s agent debugger documentation lists what text mode misses: a name transcribed with a different spelling, digits transcribed as words, and an endpointing error, where a turn ends before the user finishes speaking.
Cekura’s voice runs simulate what real callers bring. In a cekura.tests.json spec, a scenario’s personality sets the caller’s speed, background_noise and a prompt such as “You interrupt often and speak quickly.”
Per Cekura’s voice agent workflow benchmark (8 configurations, 82 scenarios, 3 retained repeats, 246 calls each), infrastructure-clean calls ranged from 72.4% to 100.0%, and calls that never connected stay in the denominator. Retell ranked first on repeatable reliability, passing every applicable rubric gate on all three runs for 75.6% of scenarios, against 93.9% task completion per call. Providers chose their own models, speech components and settings, so each result describes the configuration that provider submitted.
How do you test a voice AI agent from Cursor, VS Code or Claude Code over MCP?
The Cekura MCP server is a remote tool server that lets the coding assistant in your IDE list agents, run scenarios and read scored results in the chat where it edits your agent. It ships with Cekura Skills in one plugin per client, signed in over OAuth (setup guide).
- Install the plugin. In Claude Code, run
/plugin marketplace add cekura-ai/cekura-skills,/plugin install cekura@cekura-skills, then/setup-mcp. In VS Code, add the marketplace from the Claude Code chat panel; Cursor imports it under Settings, Plugins. - Match the transport. Each
scenarios_run_*tool drives one connection, so the assistant reads the agent’sprovider.typeand telephony block first. - Start with a low frequency (runs per scenario), then raise it once a run succeeds.
- Triage next to the code. The assistant compares the failed transcript with your prompt and codebase, proposes a fix, and reruns that evaluator.
Voice runs bill per minute and large transcripts can fill the context window, so page through results. The design of Cekura’s MCP tools covers OAuth sign-in and tool descriptions.
How do you test LiveKit or Pipecat voice agents locally from your IDE?
LiveKit’s agent debugger is a local harness that lets a coding agent such as Claude Code play the user one turn at a time and read the tool calls behind each reply. Pipecat Evals runs scenarios against your real Pipecat pipeline with one command.
- LiveKit, try the change.
lk agent debugger start,lk agent debugger say "Book me a table for two tonight", thenlk agent debugger restartafter every edit, since a running session keeps old code (CLI v2.19.0 or later). - LiveKit, pin the bug as a pytest or Vitest test on messages, tool calls and handoffs.
- LiveKit, simulate before pushing.
lk agent simulate text --scenarios scenarios.yamlexits non-zero on a failed scenario and needs a LiveKit Cloud project. - Pipecat, run the evals. With the bot running,
pipecat eval run scenarios/exits 0 or 1 and writes<scenario>.eval.log, which Pipecat’s eval loop guide says to read before changing code.
Pipecat’s evals lifecycle guide states the limit: the local eval transport is not a real WebSocket, SIP or telephony path.
How do you run voice agent tests from your IDE terminal with a CLI or SDK?
The Cekura CLI is a terminal tool that ships with the Python SDK in pip install cekura, for Python 3.9 or later. It suits an assistant that prefers shell commands to MCP tools (CLI reference).
- Sign in with
cekura auth login, which opens a browser. Keep API keys for CI. - Run a fast check with
cekura run start --agent-id 123 --scenario-ids 1,2,3 --mode text, then--mode voicefor an outbound call through your provider. - Make it a gate with
--wait --fail-on-threshold 1.0. It exits 1 below the threshold and 2 when the run cannot start or the wait times out (default 3,600 seconds). - Keep tests in the repo as a
cekura.tests.jsonspec. Its$schemaline adds autocomplete and inline validation in most editors, andcekura run spec cekura.tests.json --agent-id "$CEKURA_AGENT_ID" --dry-runchecks it without spending anything.
Self-hosted LiveKit and Pipecat runs start from the SDK or the API instead. Running voice agent tests from Claude Code, Cursor or Codex covers the spec format and the skills that write it.
How do you tell your IDE’s coding agent to test voice AI agents with AGENTS.md?
AGENTS.md is a project file that tells any coding agent how to verify a change before calling it done. Cekura’s AGENTS.md template defines the terms agent, evaluator, metric, result and run, and leaves slots for your agent ID, project ID, provider and key metrics. Fill the slots so the assistant starts without asking for IDs, then add a recipe:
### Voice agent tests
After changing the agent's prompt, tools or pipeline:
1. LiveKit: `lk agent debugger start`, then `lk agent debugger say "..."`. Read the tool calls and reply. Run `lk agent debugger restart` after every edit.
2. Before a pull request, run `cekura run spec cekura.tests.json --agent-id <AGENT_ID> --wait --fail-on-threshold 1.0`. It places voice calls, which bill per minute, and exits non-zero unless every run passed.
3. Add a scenario for every behavior change. Never loosen a criterion to make a run pass.
npx skills add livekit/agent-skills --skill debugging-livekit-agents installs LiveKit’s debugger skill from its agent-skills repository. The template’s MCP command carries an API-key header, so keep keys out of the commit.
How do you gate voice agent tests from your IDE in CI?
A CI gate is a required pull-request check that reruns the scenarios your assistant ran in the IDE and blocks the merge when one fails. Cekura’s GitHub Action, cekura-ai/cekura-github-actions@v1.2.3, takes an agent ID, scenario IDs (or tags, for voice and text runs), and an execution_mode of voice, text, livekit_v2 or pipecat_v2.
Cekura’s result statuses guide defines what green means. Starting a run returns once runs are queued, before anything is dialed. A result reaches completed when every run has ended and at least one completed, which says nothing about passing. A run that errored or never connected counts as neither success nor failure, so zero failures is not a pass. The action passes only when the result is completed and every run passed.
Set frequency: '3' on the step that runs your merge-gating scenarios; it triples voice minutes. LiveKit’s guide to running simulations in CI puts text runs on every commit and audio runs nightly or before a release. Workflow files are in Cekura’s GitHub Actions guide.
Frequently asked questions
Can I test a voice AI agent from VS Code without Claude Code?
Yes. Cekura’s manual setup page gives an mcp-remote configuration for Cursor and VS Code, which needs Node.js 20.18.1 or higher, and OAuth signs you in through the browser. In .vscode/mcp.json, VS Code’s own MCP documentation puts that entry under a top-level servers key. GitHub Copilot users can install the Cekura plugin in Copilot CLI, while Copilot’s IDE extensions read Skills from a repository’s .github/skills/ directory. LiveKit and Pipecat local tests run in the integrated terminal either way.
Can I test a voice agent from my IDE without paying for calls?
Partly. LiveKit’s debugger in text mode costs only your agent’s own LLM and tool calls, and Pipecat’s text mode skips STT and TTS usage, but neither reaches the deployed phone or WebRTC path. Cekura’s pricing page lists $0.25 per voice testing minute and $0.025 per reply, and new accounts start with 300 free credits, about 60 minutes. Dry-run a committed spec first to see the estimated cost.
Can a coding agent test a voice agent that only runs on my laptop?
Yes. In text mode, LiveKit’s debugger launches the agent on your machine without a LiveKit room, and Pipecat Evals connects to the bot’s local development runner. A Cekura chirp run reaches an agent that exposes a raw-PCM WebSocket endpoint (16 kHz, 16-bit mono), including one behind a tunnel: chirp_data.websocket_url passes a fresh ngrok or Cloudflare Tunnel URL for each run without editing the agent.
How do I stop a coding agent from weakening its own voice agent tests?
Treat scenario files as reviewed code. Pipecat’s guidance warns that an assistant able to edit scenarios can loosen an eval: criterion, a success condition or a metric threshold, and that failing evals should usually be fixed in the agent. Give each spec case a stable key so results stay comparable, and require every run to pass in CI. Cekura’s guide to regression testing with a coding agent covers separating the writer from the grader.
Does Cekura work with the LiveKit and Pipecat tests I already run in my IDE?
Yes. Cekura keeps your local tests and adds runs over the deployed path. Cekura creates a LiveKit room and token for each scenario on the livekit_v2 channel, starts Pipecat Cloud sessions on pipecat_v2, and also dials over telephony or SIP. Pipecat’s own docs have a page on Cekura for Pipecat agents, and the LiveKit voice agent testing page covers the LiveKit side. To see it on your own agent, book a demo.
