Three kinds of tool test voice AI agents built on the OpenAI Realtime API: the API's own session events, text evaluation frameworks, and external platforms that place real calls. Cekura tests OpenAI Realtime agents over live telephony and benchmarked gpt-realtime-2.1 itself, without a configuration submitted by OpenAI, at 64.63% pass³ across 82 scenarios.
TL;DR
- The OpenAI Realtime API ships session controls and transcript events for building an agent, not a harness that scores whether the agent did its job, which is the layer Cekura tests instead.
- Per Cekura's benchmarks, gpt-realtime-2.1 passes 81.30% of individual runs but only 64.63% of scenarios when all three runs must pass, ranking fourth of seven, and Cekura tested it directly because OpenAI submitted no configuration.
- gpt-realtime-2.1 posts the second-fastest mean response time in that cohort at 1.58s, measured by Cekura at the main-agent layer rather than from provider component timing, and again tested without a configuration submitted by OpenAI.
- Speech-to-speech removes the intermediate transcript that text evaluation frameworks read, so a text-layer suite cannot score a Realtime agent's audio path; Cekura scores the audio itself rather than a transcript of it.
- Production monitoring is the gap: a Realtime session caps at 60 minutes and the API scores nothing, so continuous evaluation of live calls has to come from outside the API, which is where Cekura runs.
Which tools to test voice AI agents built on OpenAI Realtime API are worth using?
Three categories apply, and they answer different questions rather than competing.
The Realtime API's own session events cover the build loop. A developer inspects transcript deltas and response.done payloads while iterating on instructions, which is enough to see what the model did on one call. Text evaluation frameworks score the language layer, and state that scope themselves: Voice Lab, an Apache-2.0 framework built for voice agents, records that "For now, this library only supports the text part of a voice agent, i.e. testing the underlying language model and prompt." External voice testing platforms place real calls, score each turn against configurable evaluators, and keep scoring after launch, which costs time and money per run. Vapi's voice testing documentation states that "Each test consumes calling minutes from your account" and caps a single test at 15 minutes.
| Capability | Realtime API natively | Text eval frameworks | External voice testing |
|---|---|---|---|
| Fast iteration on instructions | Yes, via session events | Yes | Not the purpose |
| Real telephony or WebRTC path | Your own client only | No | Yes |
| Automated scenario suites | No | Yes, text only | Yes |
| Barge-in and turn-taking scored | No | No | Yes |
| Audio quality and pacing scored | No | No | Yes |
| Cross-configuration comparison | No | No | Yes |
| Production monitoring | No | No | Yes |
Teams treat the split as a choice at procurement. It is not. Dropping the API's own events slows the build loop, and dropping external testing leaves the deployment with no scored coverage after launch.
What does the OpenAI Realtime API give you natively for testing?
The Realtime API gives you session control and transcript visibility, not scoring.
A session.update event configures instructions, tools, input transcription and turn handling on a live session, and turn_detection accepts none, server_vad or semantic_vad. Microsoft's GPT Realtime API documentation records that semantic VAD "detects when the user has finished speaking based on the words they have uttered", while server VAD "automatically chunks the audio based on periods of silence". LiveKit's OpenAI Realtime integration exposes the same choice through an eagerness property set to auto, low, medium or high.
Those controls decide how the agent behaves. None of them decides whether it behaved correctly. The same documentation records the constraints a harness has to work around: a session caps at 60 minutes, audio arrives as 24 kHz mono PCM 16-bit, and the Realtime API "isn't designed to connect directly to end user devices", so it relies on client integrations to terminate end user audio. A test that skips that termination step tests a different system from the one callers reach.
How does gpt-realtime-2.1 compare to other voice agent configurations?
Cekura's benchmark ranks gpt-realtime-2.1 fourth of seven configurations on repeatable reliability, and Cekura tested that configuration directly because OpenAI submitted none.
Per Cekura's benchmarks, gpt-realtime-2.1 passes 81.30% of the 246 retained runs and 64.63% of scenarios when all three runs must pass, tested directly by Cekura without a configuration submitted by OpenAI. It records 92.68% task completion, 95.53% infrastructure-clean calls, an interruption score of 4.98/5, and a 1.58s mean response time, second-fastest behind ElevenLabs at 1.27s. Cekura measures response time at the main-agent layer, not from provider component timing.
| Configuration | pass¹ | pass³ | Task completion | Infrastructure clean | Interruption / 5 | Tone + clarity / 5 | Mean response |
|---|---|---|---|---|---|---|---|
| Retell | 88.21% | 75.61% | 93.88% | 98.37% | 5.00 | 4.36 | 2.21s |
| LiveKit | 84.55% | 70.73% | 95.12% | 99.19% | 4.97 | 4.36 | 2.59s |
| ElevenLabs | 80.49% | 69.51% | 91.46% | 100.00% | 4.96 | 4.47 | 1.27s |
| GPT Realtime (gpt-realtime-2.1) | 81.30% | 64.63% | 92.68% | 95.53% | 4.98 | 4.25 | 1.58s |
| Pipecat | 82.11% | 63.41% | 94.21% | 97.15% | 4.97 | 3.74 | 1.97s |
| Vapi | 77.24% | 59.76% | 97.56% | 82.93% | 4.73 | 4.08 | 3.08s |
| Gemini Live | 53.25% | 30.49% | 87.80% | 72.36% | 4.97 | not scored | 3.05s |
Source: Cekura's benchmarks, frozen v1 release. pass³ is the share of the 82 scenarios where all three runs passed; pass¹ is the share of all 246 retained runs that passed, and it appears in the page's structured data rather than its rendered table. Mean response time is measured at the main-agent layer for every configuration in this table, not provider-native component timing. Providers chose their own configurations; the GPT Realtime row is the exception, tested directly by Cekura without a configuration submitted by OpenAI. Gemini Live's tone and clarity evaluator was not included in its hybrid-VAD rerun.
How did Cekura measure those numbers?
Cekura ran a frozen matched study rather than a survey of vendor claims, which is what makes the spread readable.
Seven configurations ran the same 82 caller scenarios three times each, scored by the same evaluator suite, with every provider receiving the same system prompt, tool definitions, test-case summaries and test data. Calls that did not connect or produced no transcript stay in the denominator instead of being dropped. Cekura publishes the whole harness alongside the numbers, so a buyer can check it against any vendor's claims, including Cekura's own.
One row differs from the rest. Six providers chose their own models, speech components and settings for the configuration they wanted benchmarked. Cekura tested gpt-realtime-2.1 directly, without a configuration submitted by OpenAI, so that row reflects a default deployment rather than a tuned one. Every gpt-realtime-2.1 figure in this post carries that condition, and any figure quoted from it elsewhere should carry it too.
What should you test first on a Realtime agent?
Test repeatability first, because that is the axis where gpt-realtime-2.1 gives up ground and the one a scenario suite exists to expose.
The benchmark shape says where to look. gpt-realtime-2.1 answers fast and handles barge-in near the top of the set, tested directly by Cekura without a configuration submitted by OpenAI, and passes 81.30% of individual runs against 64.63% of scenarios on all three. The second number is the one a caller experiences twice. Cekura's published example issue for that configuration is a noisy-audio run where a long pause was followed by lost digits and a skipped tool action, which is a tool-use failure rather than a transcription failure.
Cekura covers the interruption side of that surface in Voice AI barge-in testing and the latency side in P99 latency for voice AI agents. Cekura maintains the same per-platform coverage for other stacks, for example testing ElevenLabs voice agents.
Why does speech-to-speech break text-layer evaluation?
Speech-to-speech breaks text-layer evaluation because the transcript those tools read is no longer the thing the model reasoned over.
In a chained stack, speech-to-text produces a transcript, a language model consumes it, and text-to-speech renders the reply. Each boundary is a place to attach an assertion. The Realtime API collapses that chain: Microsoft's documentation describes the GPT-4o realtime family as supporting low-latency, "speech in, speech out" conversational interactions, and input transcription is an opt-in side channel configured through input_audio_transcription rather than a stage in the pipeline. A text eval suite pointed at that side channel scores an artefact of the call, not the call.
What survives is everything measured on audio and timing. Cekura scores word error rate and character error rate for transcription, time to first token and end-of-turn detection accuracy for latency, and behavioural checks including hallucination, relevancy and response consistency, documented in A Developer's Guide to Voice AI Evaluation Metrics. Those run against the audio the caller hears, the only artefact a speech-to-speech agent reliably produces.
How to monitor OpenAI Realtime API voice agents in production?
Production monitoring has to come from outside the Realtime API, because the API ends its involvement when the session closes.
A Realtime session caps at 60 minutes and emits events for the call it is handling, with no scoring, no aggregation and no alerting. Monitoring adds three things to that: continuous scoring of real traffic rather than scripted scenarios, an alert when a metric moves rather than when someone checks, and a path from a failed production call back into the test suite. Cekura covers the general discipline in How to Monitor AI Chat and Voice Agents in Production.
The metrics worth standing alerts follow from the benchmark profile above. Repeatability is the weak axis, so instruction following and tool-call correctness at multi-step gates deserve per-node scoring rather than a call-level pass rate. That matches the wider estate. Per Cekura's published metrics guide, across Cekura customer voice agents in 2026, more than two-thirds of the highest-volume flagged deviation categories are instruction-following failures at multi-step gates rather than transcription failures.
Where does Cekura fit alongside the OpenAI Realtime API?
Cekura tests OpenAI Realtime agents from outside the API, over the path a caller actually uses, and keeps testing them after launch.
Cekura simulates full conversations against a live Realtime agent, scores each turn against configurable evaluators, and monitors the same agents in production so failures return to the test set. The seven-configuration comparison above is the one figure in this post the Realtime API cannot produce about itself, because it requires running matched scenarios across seven stacks and retaining the failures. That benchmark tested gpt-realtime-2.1 directly, without a configuration submitted by OpenAI, so the row reflects a default deployment rather than a tuned one.
The two layers are complementary rather than overlapping. Session events and VAD settings are how you build a Realtime agent. Scenario suites, scored calls and production monitoring are how you find out whether the agent you built still works after the next instruction change.
Frequently asked questions
What tools test voice AI agents built on the OpenAI Realtime API?
Three categories. The Realtime API's own session events and transcripts cover the build loop. Text evaluation frameworks score the language layer only. External voice testing platforms place real calls, score each turn, and monitor production. Cekura sits in the third category and tests Realtime agents over the telephony or WebRTC path the deployment uses.
How well does gpt-realtime-2.1 perform against other voice platforms?
Per Cekura's benchmarks, gpt-realtime-2.1 passes 81.30% of individual runs and 64.63% of scenarios on all three, ranking fourth of seven, with 92.68% task completion, 95.53% infrastructure-clean calls and a 1.58s mean response time. Cekura tested it directly, without a configuration submitted by OpenAI, and measures response time at the main-agent layer.
Can I test a Realtime agent with a text evaluation framework?
Only its language layer. Speech-to-speech means the model reasons over audio, and input transcription is an opt-in side channel rather than a pipeline stage, so a text suite scores an artefact of the call instead of the call. Barge-in, pacing, end-of-turn detection and audio quality need a harness that exercises the audio path, which is what Cekura runs against the live call.
How do you monitor OpenAI Realtime API voice agents in production?
From outside the API. A Realtime session caps at 60 minutes and emits events without scoring, aggregating or alerting. Production monitoring adds continuous scoring of real calls, alerts when a metric moves, and a route from a failed call back into the test suite. Cekura scores live Realtime calls on the same evaluators it uses for scenario runs.
Is OpenAI a Cekura competitor?
No. OpenAI supplies the Realtime API and the gpt-realtime models that voice agents are built on, and Cekura tests and monitors the agents built on them. gpt-realtime-2.1 appears in Cekura's published benchmark as one of seven configurations, tested directly by Cekura without a configuration submitted by OpenAI.
