TL;DR: Voice AI simulation means running a realistic simulated caller (a "testing agent") through full conversations with your AI agent before a real customer ever does. The value of every metric downstream depends on how faithful that simulation is. Get the simulator's accuracy and latency wrong, and every result built on top of it is meaningless.
You can't trust an AI agent you've only tested by hand. Every conversational AI agent, the one that answers your support line, books your appointment, or handles your insurance claim, is only as good as the situations it has been tested against. But testing a conversational agent is different from testing ordinary software. There is no fixed input and expected output. There is a conversation: open-ended, branching, and shaped moment-to-moment by whoever is on the other end.
You can't hire hundreds of people to call your agent every time you change a prompt. The answer is voice AI simulation: an automated testing agent that plays the role of a realistic customer, holds a full conversation with the agent under test, and lets you evaluate what happened afterward. It's the engine underneath agent testing. And if the simulated caller behaves like a real person, your results mean something. If it doesn't, every metric downstream is built on sand. At Cekura, this is the problem we spend the most engineering time on.
What We Mean by "Simulation"
In every simulated run there are two participants:
- The main agent — the customer's agent, the thing being tested. We treat it as a black box.
- The testing agent — our simulator. It plays the caller, driven by a model, given a script and a persona, carrying on the conversation turn by turn.
The simulator's behavior comes from two inputs:
- The scenario — an ordered script of what the caller is trying to do: "ask about a bill, dispute a charge, refuse to share your phone number, hang up once it's resolved." This is the deterministic behavior we actually want to test.
- The persona — tone, verbosity, accent quirks, how they read out a number or a zip code. When the two conflict, the script always wins: the persona is a costume, not a rewrite of the plot.
Crucially, the simulator doesn't just produce text. On every single turn it makes three decisions at once: what to say next, whether it's time to end the call, and whether the conversation has gotten stuck in a repetitive loop. Bundling those into one structured response is deliberate: it's both more reliable and faster.
For more on how these scenarios are built and run, see Conditional Actions: Robust Testing of Chatbots and Voice Agents.
Why Accuracy Is Everything in Voice AI Simulation Testing
A simulation is a stand-in for reality, and the whole value of a test depends on the simulated caller behaving the way a real one would. A simulator that's too cooperative (never mishearing, never pushing back) hides real bugs. One that's too erratic invents failures that will never happen and sends your team chasing ghosts. Either way, trust in the results is gone.
Much of the difficulty is in timing and cues: a faithful simulator has to wait for the agent to actually reach a point in the conversation before it reacts, rather than jumping ahead. A common, subtle failure is preemption. The script says "when the agent asks for your date of birth, provide it." And the simulator volunteers the date of birth before the agent has asked, just because the script mentions it. The trigger hasn't fired yet, but the simulator fires anyway, and now the transcript shows a caller answering a question that was never posed. That's something no real caller would do. Getting this right means the simulator must treat each scripted step as gated on its own specific cue and hold back until the agent genuinely gets there.
That's why building an accurate voice AI simulation is an empirical, iterative discipline, not a one-time prompt-writing exercise. We test the model and the prompt across many scenarios, personas, and reproduced real calls; find the classes of conversation where the simulator jumps ahead, ends too early, or lingers too long; and tighten the instructions with general rules (never scenario-specific patches) until its judgment matches a human's.
Why Latency Is a Hard Constraint
On a live voice call, the agent under test is waiting on the line for the caller to respond, and every extra second the simulator spends thinking is dead air. Real callers don't pause for four seconds before saying "yes, that's right"; an agent that hears silence often reacts to it, re-prompting with "Are you still there?" or repeating itself. A slow simulator doesn't just make a slow test; it makes a different, unrealistic conversation.
So we keep the simulator fast:
- Minimal or no extended "reasoning" between turns
- A cap on how much it says per turn
- All of its decisions made in a single request
- Caching of the large, stable part of the prompt
It's a deliberate trade-off: we add latency back only where it protects accuracy, such as a quick second-opinion check before actually hanging up, so the simulator doesn't end a call early on a false signal.
For more on measuring these tradeoffs in practice, see Performance Testing for Voice Agents: A Practical Guide.
Making a Fully Synthetic Call Feel Real
A simulated conversation has a hidden problem the transcript never shows: there's no real customer, no real account, and no real backend behind the agent. Three features work together to close that gap:
| Feature | What It Does |
|---|---|
| Mock tools | Real agents call external systems mid-conversation: look up an account, check a balance, book a slot. In a test, we don't want those calls hitting production systems, and we can't have a test fail just because a third-party API was flaky. So we intercept the agent's tool calls and return realistic, pre-defined responses instead. The agent believes it fetched real data and the dialogue proceeds naturally past the tool step, deterministically and repeatably. Afterward, we can also check whether the agent called the right tool with the right arguments. |
| Dynamic variables | These are the main agent's variables: the per-call data the agent under test is handed for this specific conversation. A support agent doesn't operate in the abstract; it's given the customer record, account ID, or plan details it's meant to act on. Injecting them per run means the agent personalizes correctly and has concrete data to work from, instead of talking to a nameless, dataless user. |
| Test profiles | A test profile holds the testing agent's variables: who the simulated caller is and the data they carry (name, date of birth, the account number they'll read out), kept separate from the scenario script. Because it's swappable, one scenario becomes a whole parameterized suite (new customer versus long-tenured, valid versus invalid account, different regions and languages) without rewriting it. |
The two sides have to line up. The payoff comes when the caller's data (the testing agent's variables) matches what the main agent knows (its dynamic variables) and what its mocked tools return. When the simulated caller says "my account is 4471," the agent's dynamic variables and mock-tool data should recognize it; otherwise a verification step fails and the call derails on an error no real customer would hit.
Text vs. Voice: Why Voice AI Simulation Is Harder
Everything above applies to chat and voice alike, but voice adds a class of problems text never has. A text conversation is a clean, discrete, untimed log: one side sends a complete message, the other replies. A voice call is a single, continuous, lossy audio stream shared in real time, so most of our voice-specific work goes not into generating replies but into reconstructing "who said what, and when."
| Problem | Why Voice Makes It Harder |
|---|---|
| Interruptions and barge-in | Both speakers can talk at once, so the system has to infer from overlapping audio whether it was a clean hand-off, a successful interruption, or the agent talking straight through one. Guess wrong and you corrupt the transcript that everything else depends on; we also score how quickly an agent yields when the caller cuts in. |
| Turn-taking | Text has an explicit "send"; voice only has silence, and silence is ambiguous: finished, or just thinking? We estimate endpoints with voice activity detection, but the common failure is the simulator answering a sentence that was actually cut off mid-thought. |
| Dead air and timing | In text a slow turn is just slow; in voice, silence is itself a failure state, and timing becomes semantic: the gap between turns, and the millisecond delay before an agent yields, are both things we measure. |
For a deeper look at voice-specific accuracy challenges, see AI Voice Agent Accuracy Testing.
Why This Matters
It's tempting to think of the simulator as plumbing: a way to generate transcripts so the "real" work of evaluation can happen. It's the opposite. The simulator is the part of the system that has to be most faithful to reality, because everything downstream inherits its assumptions. An evaluation metric can only judge the conversation it was given; if that conversation wasn't realistic, the judgment is meaningless.
Building a good voice AI simulation means holding two things in tension: it has to be accurate enough that its behavior is indistinguishable from a real caller's, and fast enough that it never distorts the live call by making the other side wait. Achieving both, across text and voice, thousands of scenarios and personas, with interruptions, misheard words, and ambiguous silence, is what makes testing conversational AI hard. Get it right, and you can test an agent against the messy reality it will actually face, long before a real customer ever does.
That's the whole point of building a realistic testing agent: to find the failures in simulation, so they never happen in production.
FAQ
What exactly counts as a "voice AI simulation"?
It's the practice of using an automated program to hold realistic spoken conversations with your voice agent before real customers do, so bugs surface during testing instead of in production.
Why not just call the agent myself a bunch of times?
Manual testing doesn't scale, you can't hire people to re-call your agent every time you change a prompt. Voice AI simulation runs the same scenario across many personas, languages, and edge cases automatically and repeatably, which manual spot-checks can't match.
Scenario vs. persona: what's actually the difference?
The scenario is the deterministic script of what the caller is trying to accomplish (dispute a charge, book an appointment). The persona is how they say it: tone, accent, verbosity. When they conflict, the scenario always wins.
Does any of this apply if I'm testing a chat agent instead of voice?
The core ideas, a program playing a realistic user, driven by a scenario and persona, apply to both. Voice adds extra problems text doesn't have, like interruptions, turn-taking, and ambiguous silence, which is why voice AI simulation requires more specialized engineering than chat.