Teams testing voice AI agents built on Amazon Connect / Lex use three layers: the Amazon Lex Test Workbench for intent and slot accuracy, the Amazon Connect test case APIs for contact flow logic, and Cekura for real calls that measure latency, barge-in and answer quality end to end. No single layer covers the other two.
TL;DR
- Amazon Lex ships its own evaluation harness, the Test Workbench, which scores intent recognition and slot resolution against reference transcripts and imports test sets of up to 50k of text or up to 5 hours of audio.
- Amazon Connect added test case APIs on February 2, 2026, covering contact flow logic, expected responses and business conditions such as after-hours scenarios, in eleven AWS Regions.
- Neither native layer measures response latency percentiles, barge-in behaviour under real audio, or whether a generated answer was correct, so those need a call-level testing platform.
- A default Amazon Connect instance allows 10 concurrent active calls, which caps load testing before you write a single test case.
- Orchestration choice moves conversational reliability independently of the model: in Cekura's voice orchestration benchmarks, one unchanged agent scored 96.6% pass^3 on the strongest platform and 76.3% on the weakest, while the lowest p50 latency belonged to the lowest-scoring platform. Those six platforms are Retell, Vapi, Pipecat, LiveKit, Synthflow and ElevenLabs; Amazon Connect and Amazon Lex are not in that dataset, and the figures are platform defaults.
What tools test voice AI agents built on Amazon Connect / Lex?
An Amazon Connect / Lex voice agent is a two-service stack in which Amazon Connect owns the telephony and the contact flow, and Amazon Lex owns speech recognition, intent classification and slot filling. Testing it means testing three distinct surfaces, because a bug in the bot, a bug in the flow, and a bug that only appears over a phone line all present differently. Amazon publishes a native tool for the first two surfaces and nothing for the third.
The separation is real, not theoretical. A Lex bot is portable across telephony fronts: AWS and Cisco published an open source Amazon Lex connector for Webex Contact Center in April 2026 that runs the same Lex bot behind Cisco's Bring Your Own Virtual Agent framework. The bot layer and the call layer are independently replaceable, so they need independent tests.
| Test surface | What it exercises | What it reports | What it leaves uncovered |
|---|---|---|---|
| Amazon Lex Test Workbench | Intent classification and slot resolution for single inputs and multi-turn conversations, from a CSV test set or generated from conversation logs | Intent recognition metrics, slot resolution metrics, conversation pass rates, the top 5 worst performing intents and slots, and a speech transcription pass rate for audio test sets | Contact flow routing, queues and transfers; response latency; barge-in; audio quality; whether a generative answer was factually right |
| Amazon Connect test case APIs | Contact flow logic from a named entry point, with configured caller identity, call reason, expected responses and business conditions | Test case execution records and an execution summary per run, callable from CI/CD with multiple tests in flight | Lex intent and slot accuracy in isolation; ASR performance under noise or accent; latency percentiles; audio fidelity |
| A real call over the PSTN | The whole stack as a caller meets it, including carrier behaviour, DTMF timing, endpointing thresholds and interruption handling | Whatever your testing platform measures: turn latency, barge-in success, transcript accuracy, task completion, evaluator scores | Nothing, provided the platform actually scores the conversation rather than only recording it |
Cekura operates on the third row, placing automated inbound and outbound calls to a claimed Amazon Connect phone number, navigating DTMF and IVR prompts, and scoring each turn against expected outcomes, which is the layer where the native tools stop.
How to test a Amazon Connect / Lex voice agent
The Amazon Lex Test Workbench is a native evaluation harness that scores a built bot against a reference test set before deployment. You generate a test set either by uploading a CSV built from Amazon's downloadable template, or from existing conversation logs, choosing a modality of Audio or Text. Amazon's own limit is explicit: the Test Workbench "can import text files up to 50k, and up to 5 hours of audio". Results arrive as tabs for overall results, conversation results, intent and slot results, and detailed results.
Read the Test Workbench failure table before you trust a pass rate. Amazon documents eight failure scenarios, and for six of them the documented action is "Skip other turns in the conversation". An intent mismatch, a slot elicitation mismatch, a missing back to back agent prompt, an extra back to back agent prompt, an unelicited optional slot and an unrecognised slot all truncate the remainder of that conversation's evaluation. Only slot value mismatches and transcription mismatches continue through the remaining turns. A suite that diverges early therefore reports less than it appears to.
How do the Amazon Connect test case APIs work?
Amazon Connect covers the flow layer separately. On February 2, 2026, AWS announced APIs to test and simulate voice interactions, available in Asia Pacific (Mumbai), Africa (Cape Town), Europe (Frankfurt), US East (N. Virginia), Asia Pacific (Seoul), Europe (London), Asia Pacific (Tokyo), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney) and Canada (Central). The announcement states you can configure the caller's phone number or customer profile, the reason for the call, the expected responses, and business conditions like after-hours scenarios or full call queues.
The API surface in the Amazon Connect API Reference is eleven operations, including CreateTestCase, StartTestCaseExecution, StopTestCaseExecution, ListTestCaseExecutions, ListTestCaseExecutionRecords and GetTestCaseExecutionSummary. CreateTestCase is a PUT to /test-cases/{InstanceId} and its EntryPoint object takes a FlowId plus a SourcePhoneNumber and DestinationPhoneNumber for voice. Two details matter operationally: the test body is an opaque JSON string in a required Content field, and a test case saved with Status of SAVED skips content validation while PUBLISHED triggers it. Save without publishing and you have stored an unvalidated test.
Why is my Amazon Connect / Lex voice agent slow to respond?
An Amazon Connect / Lex agent's response time is set by contact flow configuration before it is set by model speed, and the defaults are tuned for form filling rather than conversation. Cekura measures the resulting turn latency on a live call, which is the only place these settings become visible.
The third layer exists because Amazon Connect's own Get customer input block exposes timing controls whose defaults decide whether callers get interrupted, and none of them appear in a Test Workbench score. End Silence Threshold, how long Amazon Lex waits after the caller stops speaking before treating the utterance as finished, defaults to 600 milliseconds, and that wait is spent on every turn before intent recognition begins, underneath whatever the bot and any Lambda add on top. Start Silence Threshold defaults to 3000 milliseconds, but Amazon documents it as the wait before assuming the customer is not going to speak at all, so it shapes abandonment rather than per-turn latency. Max Speech Duration defaults to 12000 milliseconds with a documented maximum of 55000 milliseconds, and setting it higher routes the contact down the Error branch.
What are the barge-in and DTMF defaults on Amazon Connect / Lex?
Barge-in behaviour differs by Lex version in a way that silently changes caller experience on migration. Amazon documents that for Amazon Lex, "Barge-in is enabled globally by default", while for Amazon Lex (Classic), "Barge-in is disabled globally by default" and must be switched on with a session attribute. DTMF has its own defaults: # ends an utterance, * clears it, the inter-digit timeout is 5000 milliseconds, and the per-utterance digit cap is 1024 characters, which Amazon notes "cannot be increased". Amazon also documents that digits entered too fast across a flow transfer can be dropped, so that a caller pressing 5123 without waiting may have only 23 or 3 captured.
These are all things a caller hits and a reference transcript never does. Cekura tests them by calling the number and measuring the result: timing and latency, DTMF behaviour, TTS prompt correctness, fallback accuracy, multi-turn state transitions and an instruction-following metric, using scripted test scenarios with branching paths and reusable components. For the carrier-side variables underneath all of it, jitter, packet loss and one-way delay, see our guides to telephony testing and VoIP call quality. If you want the platform-agnostic method rather than the Connect-specific one, our IVR testing guide covers the seven testing types on their own terms.
Why does my Amazon Connect load test stop at 10 calls?
Amazon Connect service quotas cap concurrent test traffic well below what most load plans assume. The default quota for Concurrent active calls per instance is 10, counting calls handled by a flow, waiting in queue, handled by an agent, and outbound calls. Exceed it and Amazon documents that "contacts get a reorder tone (also known as a fast busy tone)", which a naive harness will record as a failed agent rather than a hit quota. The quota is adjustable, but AWS states a request "can take up to 3 weeks" and that "An extra-large worldwide increase can potentially take months".
AWS's own launch post sets a second ceiling below the call quota. The native testing feature supports up to five concurrent test executions per instance, so a suite that fans out test cases hits five before it hits ten, and the concurrency you can actually exercise natively is half what the call quota suggests.
What are the Amazon Connect testing quotas?
Three more ceilings shape a Connect test programme. Amazon Lex bots per instance is 70 and is listed as not adjustable at the resource level. Operations in the Amazon Connect Service API default to a RateLimit of 2 requests per second and a BurstLimit of 5, with documented exceptions, which throttles a CI job that fans out StartTestCaseExecution calls. And Amazon Lex request pricing is per input, at $0.004 per speech request, with Amazon's own worked example putting 8,000 speech requests at $32.00, so a large regression suite has a per-run cost you can compute in advance.
Plan capacity before writing scenarios, not after the first load run fails. Cekura runs suites against the concurrency you actually have and reports which failures were the agent and which were the platform refusing the call.
What does a high-volume Amazon Connect / Lex deployment test for?
"We wanted to empower Dashers to get help with their most common questions and issues as quickly and efficiently as possible, saving them time, effort, and increasing their trust in DoorDash’s self-service capabilities,"
Chaitanya Hari, Contact Center Product Lead at DoorDash
DoorDash built that agent on Amazon Connect, Amazon Lex and Amazon Bedrock Knowledge Bases. In the AWS write-up, AWS reports that "The solution fields hundreds of thousands of calls per day, responding to Dashers with answers to their questions in 2.5 seconds or less," and lists what the architecture had to provide alongside the agent itself: "It also enables operational capabilities including automated testing, conversation analytics, monitoring and observability, and LLM hallucination prevention and detection." Automated testing and monitoring are in Amazon's own reference architecture, not bolted on afterwards.
The reference implementation AWS published for that architecture ships the harness in the repository: notebooks that run a test-case workbook of 50 cases against the deployed bot and generate ground-truth answers for questions that have none. It scores answer content. It does not place a call, so it reports nothing about latency, barge-in or DTMF.
How to monitor Amazon Connect / Lex voice agents in production
Amazon Connect production monitoring is an assembly of four native feeds, each with a documented guarantee you have to design around. Contact events are a near real-time stream published through Amazon EventBridge, with event types INITIATED, CONNECTED_TO_SYSTEM, CONTACT_DATA_UPDATED, QUEUED, CONNECTED_TO_AGENT, DISCONNECTED, PAUSED, RESUMED and COMPLETED. Amazon is direct about the guarantees: "Although events are not ordered, they have a timestamp which enables you to consume the data," and "Events are emitted on a best effort basis."
What do Amazon Connect contact records retain?
Contact records are the second feed and carry the durations. Amazon delivers contact records at least once, keeps each one available for 24 months from the time the contact was initiated, and instructs consumers of the export stream to check for duplicates, using LastUpdateTimestamp to spot newer copies and ContactId to deduplicate. The DisconnectReason field standardises telephony outcomes into TELECOM_BUSY, TELECOM_NUMBER_INVALID, TELECOM_POTENTIAL_BLOCKING, TELECOM_UNANSWERED, TELECOM_TIMEOUT and TELECOM_ORIGINATOR_CANCEL. On spam blocking specifically, Amazon warns that "Amazon Connect expects to see around 1% of calls have this effect," which sets a floor on how tight an outbound failure alert can be.
Amazon Lex conversation logs are the third feed and the only one that shows what the bot understood. They are configured per bot alias, so a bot promoted to a new alias can go dark unless logging is configured again. Text logs, including transcripts of audio input and metadata such as intent name and slot values, land in Amazon CloudWatch Logs; audio logs land in Amazon S3. Contact Lens is the fourth, providing conversational analytics and evaluation forms. Amazon Connect contact search reaches back two years, but a single query spans at most 8 weeks, so a year-long regression comparison is seven queries, not one.
What does Amazon Connect not tell you about agent quality?
Two gaps survive that assembly. First, the analytics are not immediate: Amazon tells you to estimate that a post-call analytics job "usually takes about 40% of the call length", and the default quota for concurrent automated interaction analytics jobs, the self-service bot interactions, is 20. Second, and more important, none of these feeds asserts whether the agent was right. There is no native event for a misrecognised intent, a hallucinated answer or a caller who gave up. Cekura closes that gap by scoring live conversations against custom evaluators and alerting on failures rather than on disconnect codes, the pattern described in our guides to voice agent monitoring platforms and unified IVR and chatbot journey testing.
How much does the orchestration layer change reliability?
Platform choice changes conversational reliability by more than most teams budget for, and it does so independently of the model. Cekura's voice orchestration benchmarks deployed one agent unchanged on six platforms, called it over the phone with the same scenarios, and scored it with the same 59 evaluators across four categories, each run three times. Pass^3, the rate at which all three runs of a scenario pass, ranged from 96.6% down to 76.3%. P50 latency ranged from 1.73s to 3.16s, and the platform with the fastest p50 posted the lowest pass^3, so speed did not predict correctness.
Two caveats bind that result. The six platforms are Retell, Vapi, Pipecat, LiveKit, Synthflow and ElevenLabs. Amazon Connect and Amazon Lex are not in the dataset, and no figure above should be read as a Connect or Lex measurement. Second, the note on the benchmarks page applies to all of it: "these are platform defaults, expect higher peaks once you tune to your use case."
How do you test an Amazon Connect / Lex agent on the call path?
The transferable finding is the method, not the numbers. Reliability on a voice stack is a property of the whole call path, so it has to be measured on the call path. For an Amazon Connect / Lex agent that means a suite that dials the number, drives the flow, exercises the Lex intents, and scores the transcript, run on every change to a prompt, an intent, a slot or a contact flow block. Cekura runs that suite through API-based execution and GitHub-driven CI workflows so a Connect flow edit and a Lex intent edit both trigger it. Book a walkthrough if you want it wired to your instance.
Frequently asked questions
How to test a Amazon Connect / Lex voice agent
Test it in three layers. Use the Amazon Lex Test Workbench for intent and slot accuracy against reference test sets of up to 50k of text or 5 hours of audio. Use the Amazon Connect test case APIs, launched February 2, 2026, for contact flow logic and business conditions. Then place real calls to measure latency, barge-in, DTMF timing and answer correctness, which neither native tool reports.
How to monitor Amazon Connect / Lex voice agents in production
Combine four native feeds and add conversation scoring. EventBridge contact events give near real-time lifecycle events, unordered and best effort. Contact records give durations and disconnect reasons, delivered at least once and retained 24 months. Amazon Lex conversation logs, configured per bot alias, show intents and slot values. Contact Lens adds analytics. None of them reports whether the agent answered correctly, so pair them with evaluator-based scoring.
Does the Amazon Lex Test Workbench measure latency?
No. The Test Workbench reports intent recognition metrics, slot resolution metrics, conversation pass rates and, for audio test sets, a speech transcription pass rate. It has no latency metric. Response timing on an Amazon Connect / Lex agent is governed by session attributes such as End Silence Threshold, which defaults to 600 milliseconds, and can only be observed on a live call.
How many concurrent test calls can I run against Amazon Connect?
Ten, on a default instance. Concurrent active calls per instance defaults to 10 and counts calls in flows, in queue, handled by agents and outbound. Calls beyond the quota receive a reorder tone, and AWS's native testing feature caps at five concurrent test executions per instance regardless. The quota is adjustable, but AWS states an increase request can take up to 3 weeks. Suites run at the concurrency you actually have, with agent failures separated from quota refusals.
Do Cekura's benchmark numbers apply to Amazon Connect or Amazon Lex?
No. Cekura's benchmarks cover Retell, Vapi, Pipecat, LiveKit, Synthflow and ElevenLabs, and Amazon Connect and Amazon Lex are not in the dataset. The transferable finding is that the orchestration layer moves pass^3 and p50 latency independently of the model, measured on platform defaults rather than tuned configurations. Treat it as evidence for measuring your own call path, not as a Connect or Lex figure.
What does it cost to run a large Lex regression suite?
Amazon Lex charges per input rather than per session, at $0.004 per speech request, and Amazon's published example prices 8,000 speech requests at $32.00. Multiply your speech turns per scenario by your scenario count and run frequency to get a per-run figure, then add Amazon Connect telephony charges for the call minutes the suite consumes.