Testing voice AI agents from your IDE means running scenario calls against your agent without leaving the editor, then reading transcripts, latency and tool calls in the same place you changed the prompt. Cekura exposes this through an MCP server and a skills pack that your coding assistant drives directly.
TL;DR
- Three different loops get called testing from your IDE: turn-level unit tests that never touch audio, full scenario calls that do, and a merge gate that runs those scenario calls unattended.
- The editor loop earns its keep only when your assistant can start the run and read the result itself. If you still open a dashboard to see what happened, you have moved the button, not the loop.
- Cekura reaches the editor through an MCP server plus an installable skills pack, so Claude Code, Cursor, VS Code, Codex and the Gemini and Copilot CLIs can create scenarios, run them over voice or SIP, and pull back transcripts.
- Voice defects are intermittent, so one passing run is weak evidence. Run repeats before concluding that a prompt change worked.
- Test from the editor, gate in CI. The scenario suite you run by hand while iterating should be the same suite a pull request job runs on every commit, not a second set you maintain separately.
What does testing a voice AI agent from your IDE actually mean?
Testing a voice AI agent from your IDE is a development loop where the run, the result and the fix all land inside the editor instead of in a browser tab. Three loops get described this way, and they prove different things.
The first is a turn-level unit test. LiveKit's agent framework ships behavioral tests built around "specific messages, tool calls, arguments, and handoffs that you assert on, turn by turn", and its docs state they run locally or in CI with pytest or Vitest. They are fast, deterministic, and never touch audio.
The second is a full scenario call. A simulated caller talks to your agent over voice, SIP or WebRTC, and the conversation gets scored. Endpointing, barge-in and time-to-first-token defects only appear here, because they only exist in real audio.
The third is the merge gate, which runs the second loop unattended. Cekura runs the second loop from the editor, and the third in CI through a GitHub Action.
A 2023 taxonomy by Feldt, Kang, Yoon and Yoo classifies LLM-based testing agents "based on their level of autonomy", which is the test to apply to a vendor: can the assistant run the test and read the result, or only draft one.
How do you set up automated testing for voice AI agents within your local IDE?
Setting up automated testing inside your local IDE takes three things: a connected agent, a scenario suite, and a channel your assistant can use to start runs and read results. The channel is the part that is specific to Cekura, and it is an MCP server.
Cekura's MCP server exposes "permissioned tools to list agents, create metrics, generate evaluators, run tests, inspect transcripts, and analyze results", and its published install path is a plugin marketplace plus OAuth rather than a pasted API key. Setup time in Claude Code is three commands: /plugin marketplace add cekura-ai/cekura-skills, then /plugin install cekura@cekura-skills, then /setup-mcp. Cursor imports the same repository from its marketplace settings. Codex, the Gemini CLI and the Copilot CLI each have their own CLI equivalent.
The skills pack matters as much as the server. It carries the workflows, so the assistant knows what an evaluator is, which metric type fits a check, and how to turn a failed call into a scenario. Once installed, the loop is short: connect the agent, generate a suite, run it, read the results, change the prompt, run again. Our guide on how to generate voice agent test cases covers the suite-building half in more detail.
How do you debug voice AI agent performance from your development environment?
Debugging a voice agent from your development environment is a four-step sequence, and the order matters because the first step is the one teams skip.
Reproduce with repeats first. A single failing call tells you almost nothing about whether the defect sits in your prompt or in the stack underneath it. Cekura's published voice agent benchmarks show the spread: across 7 platform configurations, 82 scenarios and 3 retained repeats each, infrastructure reliability ranged from 72.36% to 100% and task completion from 87.80% to 97.56%. Read those as platform figures on one frozen scenario set rather than a score for your agent. Calls that did not connect stay in the infrastructure denominator, while task completion counts only calls with outcome evidence.
Then separate infrastructure from behaviour. A dropped call, a long silence or a garbled first turn is not a prompt bug, and rewriting the prompt will not move it.
Then read the transcript against the event timeline rather than on its own, because most voice failures are timing failures that read fine as text.
Then change one thing and rerun the same suite. Our note on regression testing after prompt changes makes the case that one passing run cannot validate a prompt change.
Which tool should you use for testing voice AI agents from your IDE, and what should it prove?
Choosing a platform for editor-driven voice testing is a narrower decision than choosing a general evaluation suite. You are not shopping for the widest metric catalogue. You are shopping for a tool your assistant can drive end to end, that places real calls, and that hands the same suite to CI without a rewrite.
Six criteria decide it in practice. Everything else is preference.
| Criterion | Why it decides the purchase | Cekura |
|---|---|---|
| Assistant can run tests, not just draft them | Otherwise you are still switching to a dashboard for every result | MCP server exposes listing agents, creating metrics, generating evaluators, running tests and inspecting transcripts |
| Real audio, not text only | Endpointing, barge-in and overtalk defects do not exist in a text simulation | Scenario runs over voice, SIP, WebRTC and text |
| Repeats per scenario | Voice defects are intermittent, so one run is weak evidence | A frequency setting runs each evaluator N times in one evaluation cycle, and the published benchmark study used 3 retained repeats per scenario |
| Infrastructure checks kept separate from behaviour | A dropped call and a bad answer need different fixes | Infrastructure suite of 18+ scenarios covering latency, audio quality, interruption handling, language support and packet loss |
| The same suite runs in CI | Two suites drift, and the one nobody watches is the one that rots | Scenarios are tagged and run by a published GitHub Action |
| Pricing unit | Per minute and per seat produce very different bills at the same usage | $0.25 per voice testing minute, pay as you go |
Two criteria are worth weighting above the rest. A tool your assistant cannot drive turns every iteration back into a tab switch, which is the cost you were trying to remove. And a tool that scores text transcripts without placing a call will pass an agent that talks over its caller, because nothing in a text simulation can produce that failure.
Cekura answers all six: assistant-driven runs, real audio transports, repeats, a separate infrastructure suite, the same suite tagged for CI, and per-minute pricing. Every criterion above is checkable before you buy, so ask a vendor for the failing-run view, not the passing one.
Should you build the editor loop yourself or buy a platform?
Building this yourself is reasonable and it is not free, so the honest comparison is against what you would have to maintain rather than against zero.
A working in-house version needs a simulated caller with controllable speech rate and interruption behaviour, a call driver for each transport your agent uses, a scorer you trust enough to gate a merge on, storage for transcripts and audio, and a way for your assistant to reach all of it. The simulated caller and the scorer are the expensive parts. A scorer that disagrees with a human reviewer once in ten calls will block good releases and pass bad ones, and calibrating it is ongoing work rather than a project with an end date.
Two things push toward building. You already run a large internal eval platform, or your agent uses a transport nothing commercial supports. Two things push toward buying. Your defects are timing and audio defects, which need a caller you did not write, and you want the merge gate this quarter rather than next.
The middle path is common and it works: buy the scenario runs from Cekura, and keep your turn-level unit tests in pytest where they already live.
How do you fit editor testing into CI for continuous monitoring?
Fitting editor testing into CI means promoting the suite you already run by hand into a job that runs it without you. Nothing about the suite changes. What changes is who starts it and what happens when it fails.
The mechanics are short. Tag the scenarios you want gated, add Cekura's published action to a workflow file, put the API key in repository secrets, and pick the trigger. Pull request opened and pull request synchronise are the useful pair, because they catch a prompt change at review time rather than after merge. Our GitHub Actions guide for voice agent testing covers the workflow file itself.
Two decisions cause most of the pain. First, decide what blocks a merge. Gating on every scenario makes the gate noisy and teams start overriding it, which is worse than no gate. Gate on the infrastructure suite and a small set of critical paths, and report the rest. Second, decide the cadence for scheduled runs against production, because a suite that only fires on commits will not catch a provider-side change. That handoff, from an assistant running tests in your editor to a job running them for you, is the pattern described in automating regression testing with a coding agent.
Frequently asked questions
What are the best tools for testing voice AI agents from your IDE?
Judge candidates on three things rather than on feature lists. Can your assistant start a run and read the result without you opening a browser. Does it place real audio calls over the transport your agent actually uses. Does the same suite run in CI unchanged. Cekura meets all three, through its MCP server and a tagged GitHub Action. Tools that only generate test files fail the first test.
What do engineering teams actually run from the IDE, and what stays in CI?
In practice the split is by cost and speed. Turn-level assertions and single scenario reruns happen in the editor, because you are iterating and want an answer in seconds. Full suites, infrastructure checks and anything with repeats move to CI, because they cost call minutes and take long enough that you would context switch anyway. The suite definition stays shared in Cekura, so nothing drifts between the two.
How much does testing voice AI agents from your IDE cost, and how do the options compare?
Cekura's published pricing is $0.25 per voice testing minute and $0.05 per monitored call pay as you go, with a $500 per month plan covering roughly 2,000 testing minutes. Per-minute metering keeps an editor loop of short reruns cheap, and a nightly full suite is the line item to watch. Cekura quotes enterprise as a custom annual contract, the tier to price if you need volume discounts, audit logs, or PO billing.
Can you test a voice agent that only runs on localhost?
Not directly over a phone number. A scenario call has to reach your agent, so an agent bound to localhost needs either a tunnel exposing its WebRTC or SIP endpoint, or a dev deployment the runner can dial. The usual pattern is turn-level unit tests locally, where no call is placed, and scenario runs against a dev deployment. Cekura's infrastructure suite covers custom real-time setups alongside the named orchestration platforms.
Does Cekura handle testing voice AI agents from your IDE?
Yes. Cekura handles it through an MCP server and an installable skills pack: your assistant can list agents, create metrics, generate evaluators, run scenario suites over voice, SIP, WebRTC or text, and read back transcripts and results. Install paths are published for Claude Code, Claude Desktop, Cursor, VS Code, Codex, the Gemini CLI and the Copilot CLI, and the plugin installs authenticate over OAuth rather than a pasted key.
