New: Cekura Voice AI BenchmarksView results

Function calling and tool-use testing for voice agents

Shashij Gupta
Written byOCT 24, 20269 MIN READ
Shashij GuptainExpert verified
Co-founder & CTO, Cekura

Has stress-tested 5M+ voice agent minutes at Cekura.

Function calling and tool-use testing for voice agents

Function calling and tool-use testing for voice agents verifies that each spoken request becomes the right tool call, with the right arguments, at the right moment, and that the agent reports the real result. Cekura runs these tests as simulated calls against mock tools that return success, error or slow responses, then scores the tool calls themselves.

TL;DR

  • Wrong argument values were the largest category of audio-induced tool-call failures in “From Text to Voice” (Laskar et al., TMLR, 2026), which converted the Confetti benchmark to synthetic speech: 57.2% for Gemini-3.1-Flash-Live and 54.3% for GPT-Realtime-1.5. Wrong high-level tool-use decisions came second at 25.8% and 37.4%, with wrong tool selection at 11.8% and 5.5%.
  • A correct transcript does not prove a correct tool call. In Cekura’s voice agent workflow benchmark, one example call sent the tool a different phone number from the one transcribed, and another narrated a tool call and invented the result.
  • Mock tools make failures attributable. Cekura mock tools return fixed outputs, HTTP error codes and delays of up to 30,000 ms, so a failed test points at the agent, not a flaky backend.
  • One passing run proves little. On τ-bench, a 2024 text benchmark, a gpt-4o agent succeeded on fewer than 50% of tasks, and its pass^8 (all eight trials passing) fell below 25% in retail (Yao et al.).

What is function calling and tool-use testing for voice agents?

Function calling and tool-use testing for voice agents is a QA practice that checks the actions an agent takes during a call, not only its words. Cekura fetches tool call names, arguments, results and latency from VAPI, Retell, ElevenLabs, LiveKit and Pipecat test runs and attaches them to the transcript (Cekura tool call testing guide).

“In the transcript, it books the right appointment. On the phone, it talks over the caller, mishears a date of birth and goes quiet while a tool runs.” Sidhant Kabra, Co-founder & President, Cekura

A tool test answers four questions: was it the right tool, were the arguments what the caller said, did the call happen once and in order, and did the agent relay the real result?

Speech makes the first two harder. In “From Text to Voice”, a paper accepted to the journal TMLR, Confetti tool-calling scores fell 1.8 to 4.8 points from text to synthetic speech for the three models compared in both modes (Laskar et al., 2026).

How do tool calls fail on voice agents?

A tool-call failure on a voice agent is a mismatch between what the caller asked for and what the backend received or the agent reported, and it takes six recurring forms. Cekura’s Tool Call Success metric flags only calls that returned an error, so most of these forms leave a call that sounds fine.

FailureWhat it looks like on the callTest that catches it
Wrong toolCaller asks to cancel, agent calls the order lookupAssert the tool name and a list of tools that must not fire
Wrong or misheard argumentTranscript shows the right phone number, the tool receives anotherExact match on normalised argument values against the test profile
Narrated, never made“Let me book that for you”, then no tool callMatch every spoken action claim to a recorded tool call
Result ignored or inventedTool returns no slots, agent confirms a slot anywayMock a specific result and assert the agent relays it
Timeout and dead airSlow API leaves the caller in silence, or the step is skippedMock a delay and an error, then score silence and recovery
Double writeAn interruption or retry fires the booking tool twiceCount calls per write tool and expect exactly one

Cekura’s voice agent workflow benchmark publishes example calls for several modes. In the Retell configuration, second on pass³ (all three retained runs passed every applicable rubric gate) at 75.6%, the transcript captured a phone number correctly but a different number was sent to the tool. An ElevenLabs call narrated a tool call and continued with an invented result. Each is a single example call, not a failure rate, and providers chose their own models, speech components and settings. Hallucination detection for voice AI covers invented results.

How well do voice AI models handle tool errors and caller corrections?

Tool-error handling is a voice agent’s ability to keep a call on track when a backend call fails, and it varies widely by model. In Cekura’s speech-to-speech benchmark, 12 realtime models ran the same Pipecat agent, prompt and tools across 82 scenarios, three runs each. On the “Tool or system errors” calls the average pass rate was 88%: five models scored 100% and GPT Realtime 2.1 Mini scored 25%, with a note that it “rarely recovers from a tool error”. Calls where the caller corrects or changes their mind averaged 77%, from 96% for Grok Voice 3.0 to 38% for Gemini 3.8 Live. Saved data is a separate weakness: every saved detail matched on 94.1% of data-saving calls for the unranked cascade reference, against 59.9% to 92.0% for the realtime models. The page does not publish the scenario count per call group, so read the tool-error and correction rates as directional.

How do you test function calling in a voice agent with mock tools?

Mock tools are stand-in endpoints that return a predefined response for each tool input, which makes a tool-call test deterministic and keeps it off production systems (Cekura mock tools docs).

Cekura auto-fetch reads tool definitions from VAPI, Retell, ElevenLabs, Bland or an agent’s own MCP server and generates sample data from recent call transcripts. LiveKit agents need manual setup. Give each tool three payloads:

  1. Success: the normal output, plus edge values such as an empty slot list.
  2. Error: a status code from 400 to 599, such as a 503.
  3. Slow: a delay_ms value up to 30,000, which exposes dead air.

Cekura returns the mock entry whose input best matches what the agent sent, so pin open-ended values such as dates with test profiles. Mock data drifts from the real API unless you re-fetch it after tool changes. How to write test cases for voice agents covers scenario structure.

How do you test that a voice agent called the right tool with the right arguments?

A tool-call assertion is a rule that reads the recorded tool calls and returns pass or fail, whatever the conversation sounded like. For one scenario, write it into the expected outcome, for example that the agent must call book_appointment with the requested date and time before confirming aloud. For rules on every call, such as never calling transfer_to_human before identity is verified, write an LLM judge or Python metric.

Python suits exact checks on simulated runs, where the test profile supplies the expected value. Cekura records each tool call as a Function Call entry, so a short metric can count calls and compare digits:

import json

books = [t for t in data["transcript_json"]
         if t.get("role") == "Function Call" and t["data"]["name"] == "book_appointment"]
args = books[0]["data"]["arguments"] if books else {}
args = json.loads(args) if isinstance(args, str) and args.strip() else args
args = args if isinstance(args, dict) else {}
digits = lambda s: "".join(ch for ch in str(s) if ch.isdigit())
expected = digits((data.get("test_profile") or {}).get("phone", ""))

_result = len(books) == 1 and expected != "" and digits(args.get("phone", "")) == expected
_explanation = f"{len(books)} booking call(s), phone sent: {args.get('phone')}, expected: {expected or 'missing'}"

Idempotency needs its own scenario. LiveKit’s documentation notes that a user interruption does not cancel a running tool, and that an interrupted write can leave an operation partially complete with no way to roll back (LiveKit function tools docs). Barge in mid-booking and assert one write, not two.

Which metrics should you track for voice agent tool calls?

Voice agent tool-call metrics are the numbers that show whether actions, not answers, are correct across a suite or a week of traffic. Cekura’s voice AI evaluation metrics guide defines tool success as successful required tool calls divided by all required tool calls, on a fixed test set. Tool Call Success is narrower: it flags errors in calls that were made, so pair it with Mock Tool Call Accuracy, which also reports expected tools that were not called.

MetricWhat it measuresHow Cekura scores it
Tool errorsWhether any tool call returned an errorTool Call Success, predefined, 0 credits
Argument accuracyExpected tools called with the expected inputsMock Tool Call Accuracy, an F1-style score from 0 to 5
Hallucinated callsSpoken action claims with no matching callTool Call Hallucination metric, or a Python metric
Task outcomeThe caller’s goal was reachedExpected Outcome, 0 credits
RepeatabilityScenarios that pass on every runpass³ across three runs

Tool success and task outcome diverge, so track both: Cekura’s speech-to-speech benchmark notes “every tool call right, outcome still missed” for Gemini 3.8 Live Extended Thinking on plain bookings. Cekura’s docs list Python metrics as free and LLM judge metrics at 0.2 credits per evaluation.

How do you monitor voice agent tool calls in production?

Production tool-call monitoring is the practice of scoring the recorded tool calls in live traffic, where real backend errors and latency appear because nothing is mocked. Cekura ingests production calls through integrations for Retell, VAPI, ElevenLabs, LiveKit and Pipecat, or through an API for custom stacks, and runs Python and LLM judge metrics on them. Custom stacks must send function_call entries, or tool metrics have nothing to score.

Count, order and result-handling checks carry over from testing, but checks against expected values do not: Cekura’s Python metric docs list expected_outcome and test_profile for simulated runs only. Python metrics can also read provider execution logs, such as “Tool call failed: get_order timed out after 5s”.

  • Sample LLM judge metrics: a 30% sampling rate scores exactly 30 of every 100 live calls.
  • Check tool results for injected text, one route for prompt injection on voice agents.

Frequently asked questions

How many times should you run each tool-call test?

Run each scenario at least three times and count it as passed only when every run passes. On τ-bench, a 2024 text benchmark, a gpt-4o function-calling agent succeeded on fewer than 50% of tasks, and its pass^8, the chance a task passes all eight trials, fell below 25% in retail. Three runs triple the simulated minutes you pay for.

Can you test voice agent tool calls in text mode?

Yes for logic, no for arguments that come from speech. On VoiceAgentBench’s 6,000+ synthetic spoken queries, ASR-LLM pipelines beat end-to-end speech models, reaching up to 60.6% average parameter-filling accuracy on English (Jain et al., 2025). Cekura’s docs describe tools that fire in Retell’s native text simulation but not in Cekura’s voice tester, caused by voice activity detection cutting the agent off before dispatch or by different model configurations. Running the same evaluator in both modes isolates the cause.

Which tool-call tests should block a release?

Block a release on deterministic checks: the expected tool is called, the write tool fires exactly once, and argument values match the test profile. Run those on every pull request with Python metrics, which consume no credits, though each simulated voice call still bills per minute. Add an LLM judge check that an injected 503 or delay produces a recovery instead of an invented result, and run it nightly or pre-release at 0.2 credits per evaluation.

Does Cekura test function calling and tool use for voice agents?

Yes. Cekura captures tool names, arguments, results and latency from VAPI, Retell, ElevenLabs, LiveKit and Pipecat runs, mocks tools with success, error and delayed responses, and scores simulated calls with Tool Call Success, Mock Tool Call Accuracy, expected outcomes and Python metrics. Book a demo to see your agent’s tool calls scored.

Test your voice and chat agents with Cekura

Cekura simulates thousands of conversations before you ship and monitors every call in production — catching broken tool calls, prompt regressions, and instruction-following failures before your users hit them.

Ready to ship voice
agents fast? 

Book a demo