Common failure points in AI phone agents cluster in seven places: turn-taking, dead air during tool calls, digit capture, tool arguments, handoffs, call setup, and consent handling. Almost none of them are model quality problems. Most sit at the seams between components, which is exactly where prompt testing never looks.
Why the phone line changes the failure set
The question of why AI phone agents fail has a duller answer than most teams expect. A chat agent runs on a forgiving clock. A caller does not. On a phone call the caller hears every gap, talks whenever they like, and hangs up when the experience breaks.
Three things arrive with the phone line that a chat deployment never sees. Audio passes through a narrowband codec and a carrier network, which degrades the signal before your speech recognizer ever sees it. Keypad digits travel as signalling events rather than words. And the whole exchange runs under a hard real-time budget that a text interface simply does not have.
That is why the failure list below is mostly about boundaries. Each item is a place where two components hand something to each other and the handoff can go wrong while both components report success. Most AI phone agent failures are not a model picking the wrong words. They are two pieces of working software disagreeing quietly.
This list deliberately skips pure intent misrecognition, where the agent heard the words correctly and guessed the wrong goal. That is a prompt and language-understanding problem, it shows up in ordinary evaluator runs, and it is not a seam failure. The seven below are the ones that pass a transcript review.
| Failure point | Where the seam is | Why it survives testing |
|---|---|---|
| Turn-taking | Acoustic silence vs. finished thought | The agent recovers, just at the wrong step |
| Dead air | Turn boundary vs. tool round trip | Averages hide the slow tail callers actually hear |
| Digit capture | Codec and packet path vs. recognizer | Only breaks on degraded audio, not happy-path calls |
| Tool arguments | Transcript vs. function payload | The transcript is correct, so logs look clean |
| Handoffs | Collected state vs. transferred state | The transfer succeeds; one field is missing |
| Call setup | Dialled call vs. scored call | Calls with no transcript fall out of the denominator |
| Consent | Captured consent vs. provable consent | Nothing errors, and the exposure is legal not technical |
Failure point one: turn-taking that is too eager or too patient
Endpointing is the decision about when the caller has finished speaking. Get it wrong in one direction and the agent interrupts someone mid-sentence. Get it wrong in the other and the caller sits through a pause long enough to prompt "hello?"
Voice activity detection alone cannot make this call reliably, because silence is not the same thing as a finished thought. A caller reciting a card number pauses between digit groups. A caller thinking pauses mid-clause. Systems that treat any 500 ms of silence as end-of-turn will cut both of them off. The practical fix is semantic end-of-turn detection layered on the acoustic signal, which is the design most turn detection stacks now converge on, at the cost of added delay before the agent starts speaking.
Barge-in is the mirror problem. When a caller talks over the agent, the agent has to stop speaking, cancel in-flight generation, and treat the new audio as a real turn. Handling only part of that leaves the caller talking to a voice that keeps going, and the failure modes here are well catalogued.
What gets measured far less is what happens after the interruption. IHBench, a 2026 benchmark of post-interruption recovery, makes the gap explicit: existing benchmarks "focus on the timing of interruptions: barge-in detection, endpointing, and turn-taking dynamics. They leave unmeasured what happens after the interruption: does the agent resume the workflow at the correct step?" Across 27 audio-language model configurations, the paper reports that recovery quality depends strongly on which type of interruption arrived, and that it behaves as a largely separate capability from the rest of the model's audio performance.
So an agent can detect a barge-in perfectly and still lose the call, because it restarts the booking flow from step one.
Failure point two: dead air while a tool runs
When your agent queries a CRM, checks inventory, or verifies an account, the conversation usually stops until the request returns. On a phone call a few seconds of nothing does not read as "working on it." It reads as a dropped call.
The standards work here is unusually specific and worth borrowing. ITU-T Recommendation G.114, on one-way transmission time, states that "it is recommended to not exceed a one-way delay of 400 ms for general network planning," and that at delays below 150 ms most applications "will experience essentially transparent interactivity." Those figures describe network transmission, not agent thinking time, so treat them as the budget the network already spends before your stack contributes anything. Whatever your speech recognizer, model, and speech synthesizer add lands on top.
Two consequences follow. First, tool calls that block the turn need either a filler utterance or an asynchronous design that lets the agent keep talking while the request runs. Second, an average response time tells you almost nothing, because callers experience the slow tail, not the mean. Measuring the distribution at each layer rather than one end-to-end average is what separates a latency number you can act on from one you cannot.
Failure point three: digits that do not survive the audio path
Ask a caller for an account number, a date of birth, or a keypad selection and you have introduced one of the least visible failure points on the list.
Keypad tones are not meant to travel as audio. RFC 4733 is direct about why: "low-rate voice codecs cannot be guaranteed to reproduce these tone signals accurately enough for automatic recognition." The specification therefore carries them as named telephone events instead, and builds in redundancy because those events are packets and packets go missing. Events longer than one packetization period are updated periodically "so that the receiver can reconstruct the event and its duration if it receives any of the update packets, albeit with delay," and the last report for an event is transmitted "a total of three times". Note the conditions in that sentence: reconstruction depends on a packet arriving, and it is late when it does.
Spoken digits fail differently and just as often. "Fifteen" and "fifty" differ by one weakly stressed syllable. A caller reading a sixteen-digit number pauses in the middle, which the endpointer may read as the end of the turn. Background noise on a mobile call degrades exactly the short, low-information tokens that digits are made of.
The test that catches this is not a happy-path call. It is a digit-heavy scenario run over degraded audio, with the captured value compared against what the caller actually said.
Failure point four: a correct transcript and a wrong tool argument
This one is worth separating out because it defeats transcript review entirely.
The transcript and the tool argument are produced by different parts of the stack. Speech recognition writes what it heard. The model then decides what to pass to the function. Nothing in that architecture guarantees the two agree. A phone number can appear correctly in the transcript and arrive at the booking API with a transposed digit, and every log you check afterwards will look clean, because the transcript is right.
The same seam produces invented tool results. An agent that announces "let me look that up for you" and then continues as though a result came back has narrated a call it never made. The caller hears a confident, specific answer. There is no error anywhere.
Reviewing transcripts cannot find either failure. You have to compare the tool call payload against the conversation that produced it.
Failure point five: handoffs that drop the field they were carrying
Transfers are where accumulated state goes to die. The agent collects a reason for the call, an account number, a consent record, and a routing decision, then passes control to a human queue or another agent.
Everything the agent gathered has to survive that boundary, and the failure is almost always partial. The transfer completes. The call connects. One field is missing, so the human who picks up asks the caller to repeat what they already gave, or the record lands without the identifier that made it auditable.
Partial success is the hard part. A transfer that fails outright is easy to alert on. A transfer that succeeds while silently dropping one argument needs explicit handoff testing that asserts on the payload, not just on whether the transfer happened.
Failure point six: the call that never connects
Before any of the above can go wrong, the call has to be established. Sometimes it is not.
Session setup fails, the media path never negotiates, or the call connects and produces no audio in one direction. These failures are unglamorous and easy to under-count, because a call that produced no transcript often falls out of your reporting instead of appearing as a failure. If your dashboard computes success over calls that generated a transcript, every connection failure quietly disappears from the denominator and your success rate rises.
Failure point seven: consent and disclosure
The last failure point carries legal weight rather than just a bad caller experience.
In February 2024 the Federal Communications Commission issued a Declaratory Ruling confirming that the TCPA's restrictions on "artificial or prerecorded voice" reach AI-generated voices. Calls using them "require the prior express consent of the called party to initiate such calls absent an emergency purpose or exemption," per FCC 24-17, adopted 2 February 2024 in CG Docket No. 23-362.
That turns two agent behaviours into compliance exposure rather than quality issues. An agent that fails to make a required disclosure has a scripting bug with regulatory consequences. An agent that collects consent and then drops the consent identifier at a handoff has produced an interaction it cannot prove was consented to. The second one is failure point five wearing a different hat, which is why the seams matter more than the model.
Ranking the common failure points in AI phone agents by measured impact
Cekura runs a frozen benchmark across seven voice agent configurations, 82 caller scenarios, and three repeats of every scenario, with the same system prompt, tool definitions, and test data given to each. Providers chose their own models and speech components, and calls that did not connect or produced no transcript stay in the denominator rather than being dropped. Those two conditions matter for reading any number below.
The headline result is about repeatability. On the published leaderboard, the share of scenarios passing on all three runs ranges from about 30% to roughly 76% across the cohort, while single-call task completion for the same configurations sits between 88% and 98%. That second figure is scored only over calls that produced expected-outcome evidence, and that coverage varies by configuration, so the two bands are not computed over identical populations. The gap between them is still the finding. A configuration can complete nearly every call it is measured on and still fail one run in three of the same scenario.
That gap tells you something specific about testing. One pass of a test suite mostly measures luck. Cekura scores each scenario three times precisely because the failures on this list are intermittent by nature: an endpointer cuts a caller off on the run where they paused a beat longer, and a digit survives two runs and not the third.
The observed failures also concentrate where this article said they would. Cekura publishes one example issue per configuration alongside the scores, and four of the seven sit at the tool boundary: a phone number captured correctly in the transcript but sent to the tool as a different number, a consent record collected but its identifier omitted from the handoff, a routing decision completed but the route ID left out, and a tool call narrated with an invented result. A fifth is a connection failure, where 41 of 246 calls did not connect and stayed visible in the infrastructure figure instead of being removed.
Infrastructure reliability across the cohort ranges from about 72% to 100%, so call setup is not a solved problem you can assume away. Interruption scores, by contrast, cluster tightly between 4.73 and 5.00 out of 5, which fits the IHBench result: detecting an interruption is largely handled, and recovering from one is measured separately and much less often.
How to catch each failure point before a caller does
The tests follow from the taxonomy, and they are mostly not prompt tests.
- Turn-taking. Run scenarios with deliberate mid-utterance pauses and with the caller talking over the agent. Assert on where the flow resumes, not just on whether the agent stopped speaking.
- Dead air. Instrument each layer separately and report percentiles. Set an alert on the tail, since that is the call the caller remembers.
- Digits. Run digit-heavy scenarios over degraded audio and compare the captured value against the intended one, character by character.
- Tool arguments. Assert on the payload. Cekura evaluates the tool call against the conversation that produced it, which is the only place a correct transcript with a wrong argument becomes visible.
- Handoffs. Treat a transfer as a contract with required fields, and fail the test when a field is missing even though the transfer succeeded.
- Call setup. Keep non-connecting calls in the denominator. A success rate computed only over calls with transcripts hides your connection failures.
- Consent. Make disclosure and consent capture assertions that run on every applicable scenario, and check that the consent identifier survives to the end of the call.
Then run the whole suite repeatedly rather than once, and on a schedule rather than only before release. Cekura monitors production calls against the same evaluators used in testing, so a regression that appears after a model or prompt change surfaces as a scored failure instead of a support ticket. Most of what is on this list is intermittent, and intermittent failures are invisible to a single green test run.
If you want to know which of the seven your own agent fails, Cekura runs them as pre-release evaluators and as continuous production monitors. You can talk to the engineers who score these calls and bring a recording of one that went wrong.
Frequently asked questions
What are the most common failure points in AI phone agents?
Turn-taking errors, dead air during tool calls, digit capture, wrong or invented tool arguments, dropped fields at handoff, calls that never connect, and consent or disclosure gaps. Cekura's benchmark data shows the tool boundary is the most frequently observed of these, with four of seven published example issues sitting there. That benchmark is a frozen study in which providers chose their own configurations, so read it as a snapshot of those setups rather than a universal ranking.
Why do AI phone agents interrupt callers?
Because the endpointer treated a pause as the end of the turn. Voice activity detection measures silence, and silence appears mid-sentence whenever a caller reads out digits, thinks, or takes a breath. Semantic end-of-turn detection on top of the acoustic signal reduces this, at the cost of added latency before the agent starts responding.
How much latency can a phone agent get away with?
ITU-T G.114 recommends not exceeding 400 ms of one-way delay for general network planning, and notes that below 150 ms interactivity is essentially transparent. That is the network's budget, not your agent's thinking time, so your stack's processing adds to it. Treat anything that leaves the caller in silence past roughly a second as a defect worth an alert.
Can transcript review catch these failures?
Not the tool-boundary ones. A transcript records what was said, so a call where the transcript is correct and the argument passed to the tool is wrong reviews as clean. Cekura compares the tool call payload against the conversation, which is what makes that class of failure visible.
How many times should each test scenario run?
More than once. On Cekura's benchmark, the share of scenarios passing all three repeats ran from about 30% to 76% across seven configurations, while single-call task completion for the same setups was 88% to 98%. Read both with their caveats: providers chose their own configurations, non-connecting calls stayed in the denominator, and task completion is scored only over calls with expected-outcome evidence, whose coverage varies by configuration. The comparison is between conditions rather than a vendor ranking, but the gap is the point: a single pass mostly measures luck.
