GitHub Actions for voice agent testing

GitHub Actions for voice agent testing means a workflow file that runs a scenario suite against your voice agent on every push, pull request or schedule, and fails the job when a scenario fails. Cekura publishes a GitHub Action that starts the runs, polls them to completion, and exits non-zero on any failed run.

TL;DR

• A voice agent test job in GitHub Actions has four parts: a trigger, a step that starts remote test runs, a poll until they finish, and an exit code that decides whether the check goes red.
• Cekura's Action (cekura-ai/cekura-github-actions) needs only an agent ID, an API key and a scenario list or tag, and it fails the job if a single run fails.
• Run each scenario more than once. In Cekura's frozen benchmark, the top configuration passed 62 of 82 scenarios on all three runs, so one scenario in four failed at least one run, counting runs that did not connect.
• The published partner CI paths from LiveKit, Vapi and Retell grade text conversations; LiveKit's audio mode and Vapi's voice Simulations are opt-in. Cekura's default voice mode runs the scenarios as voice calls, billed per testing minute.

What does a GitHub Actions workflow for voice agent testing actually do?

A GitHub Actions workflow for voice agent testing is a YAML file in .github/workflows/ that hands your scenarios to a test service, waits, and turns the result into a pass or fail check on the commit. The runner starts remote runs over an API and waits for them.

Cekura's Action is a composite step built on curl, bash and python3, all preinstalled on ubuntu-latest, so the job needs no install step. Scenarios live in Cekura and are selected by ID or tag, so the workflow file carries the selection and Cekura carries the scenario definitions. Per the Cekura GitHub Actions repository, it posts the agent ID with either scenario IDs or tags, polls the result every 30 seconds, and stops at a timeout that defaults to 3,600 seconds. On completion Cekura writes total runs, successes, failures and success rate to the job summary, sets result_id and result_url outputs, and generates a results link that expires after seven days.

The gate is binary. Cekura exits the step with an error when the failed-run count is above zero, so one failing scenario fails the check. For what belongs in that gate beyond the workflow file, see CI/CD testing for voice AI agents.

Why should a GitHub Actions voice test run each scenario more than once?

One green run in GitHub Actions is one sample from a system that does not answer the same way twice. Berk Atil and co-authors at Penn State University and Comcast AI Technologies ran five LLMs configured to be deterministic on eight tasks, 10 runs each, and reported accuracy varying by up to 15% across runs. In their words, "none of the LLMs consistently delivers repeatable accuracy across all tasks" (Non-Determinism of "Deterministic" LLM Settings, preprint, April 2025). The study covers text tasks, not voice calls.

Voice agents add a second reason. Cekura's voice agent workflow benchmark is a frozen study of 8 configurations, 82 scenarios and 3 repeats, on stacks the providers chose. Its leader passed 62 of 82 scenarios on all three runs, so 20 failed at least one run. Calls that did not connect count as failures there, so the 20 mixes intermittent, repeatable and infrastructure misses.

Cekura's Action exposes this directly through the frequency input, which runs each scenario N times. Set it to 3 on the merge-gating suite. It triples the voice minutes, and a scenario that passes only sometimes fails before merge instead of in production. Regression testing for voice AI agents applies the same all-three-runs rule.

How do you set up automated GitHub Actions for voice agent testing in a CI/CD pipeline?

Setting up GitHub Actions for voice agent testing with Cekura takes a secret, a variable and one workflow file. Store the API key as the repository secret CEKURA_API_KEY, store AGENT_ID as a repository variable, then commit this file, which hardcodes the tag so each trigger can carry its own:

name: Voice agent tests
on:
  pull_request:
jobs:
  voice-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: cekura-ai/cekura-github-actions@v1.1.0
        with:
          agent_id: ${{ vars.AGENT_ID }}
          tags: smoke-test
          frequency: '3'
          api_key: ${{ secrets.CEKURA_API_KEY }}

Pin the version tag. Cekura's LiveKit and Pipecat examples use v1.1.0: set execution_mode to livekit_v2 or pipecat_v2 and pass scenario_ids, since those modes accept neither tags nor phone_number. Cekura reads the platform credentials already saved on the agent, so no LiveKit or Pipecat keys go in the workflow. For outbound agents, phone_number supplies the number to call.

The Action is a wrapper over Cekura's run and results endpoints. Teams that want another CI system can call the same programmatic voice agent testing API directly.

Which tools run voice agent tests from GitHub Actions, and how do they compare?

The tools that run voice agent tests from GitHub Actions split on one question: whether the job grades a text conversation or a voice call. Text runs are faster and cheaper. Voice runs are the only ones that reach speech recognition, turn-taking and the voice itself.

LiveKit states in its agent simulations guide that text mode runs "without the STT and TTS pipeline" and recommends it for CI. Vapi's CI/CD example checks saved staging state and "does not deploy a pull request's prompt changes". Vapi also documents Simulations, which run in chat or voice mode behind a run API that returns pass and fail counts. Both published CI examples are sound gates for prompt logic and tool-call arguments, and both are cheap enough to run on every commit. What a default text run cannot see is a misheard account number, an agent that talks over the caller, or a voice that clips. Cekura's default voice mode runs the same scenarios as voice calls, and its livekit_v2 and pipecat_v2 modes point those runs at LiveKit and Pipecat agents directly.

Should you build GitHub Actions voice agent testing in-house or buy a platform?

Building GitHub Actions voice agent testing in-house means owning five systems that are not the agent: a caller that dials or joins a room, an LLM that plays the customer, graders for each metric, a poller that survives long runs, and storage for transcripts and audio a reviewer can open from a failed check. The caller and graders carry the maintenance, because every change to your telephony provider, speech stack or prompt format has to be mirrored in the test harness.

The in-house route wins when your agent runs on a stack no platform connects to, or when your test data cannot leave your network. Its cost is a standing engineering commitment that grows with scenario count.

Cekura replaces that harness with a hosted caller, evaluators and a results store behind one workflow step. The published pricing is $0.25 per voice testing minute and $30 a month per seat after the first. As an illustration: a 20-scenario suite at frequency: '3' with calls averaging two minutes is 20 × 3 × 2 = 120 voice minutes, or $30 per workflow run. Measure your own call length. For compliance and audit requirements, the same page lists a signed BAA, DPA and 90-day log retention on the Startup plan.

What are the best practices for GitHub Actions voice agent testing workflows?

The best practices for GitHub Actions voice agent testing are about spending voice minutes where they catch regressions. Five rules cover most workflows.

1. Tag by trigger. Cekura selects scenarios by tag, so run a short smoke-test tag on every pull request and a full regression tag on a nightly schedule, as covered in automated recurring voice agent tests.
2. Make the job a required status check. A red check that does not block merge is a notification, not a gate. Add the job to branch protection.
3. Chain staging before production. Give each environment its own agent ID and API key, and use needs: so the production job runs only after the staging job passes.
4. Plan for forks. Cekura's documentation notes that forked repositories cannot read the parent's secrets, so pull requests from forks will not reach the API key.
5. Size the timeout to the suite. A tag with many scenarios at frequency: '3' can outrun the 3,600-second default. Raise timeout rather than cutting repeats.

Every Cekura run leaves a result ID and a job summary in the Actions log, which gives a change review a per-commit record of what was tested and what failed.

Frequently asked questions

What are the best tools for GitHub Actions voice agent testing?

The best tool depends on which layer you need to gate. LiveKit's lk agent simulate, Vapi Evals and Retell batch tests grade text conversations from CI and suit logic and tool-call checks; Vapi Simulations add a voice mode over an API. Cekura's GitHub Action runs the scenarios as voice calls, including against LiveKit and Pipecat agents, and fails the job on any failed run, which covers speech and turn-taking as well.

How much does GitHub Actions voice agent testing cost?

GitHub Actions voice agent testing costs the test platform's usage plus runner minutes, and runner time is small because the job only polls. Cekura publishes $0.25 per voice testing minute, with one free seat and $30 a month per additional seat. Multiply scenarios by frequency by average call length to estimate each run. Text-based partner tests bill per message instead, per each vendor's pricing.

Does Cekura handle GitHub Actions for voice agent testing?

Yes. Cekura maintains cekura-ai/cekura-github-actions, currently at v1.1.0. It takes an agent ID, an API key, and scenario IDs or tags, and supports frequency, timeout, phone_number and three execution modes: voice, livekit_v2 and pipecat_v2. Cekura writes a results summary and a shareable link to the job, and exits with an error if any run fails.

Should voice agent tests run on every pull request or nightly?

Voice agent tests should run on both, with different suites. Run a short smoke tag on every pull request so a broken prompt or tool call blocks merge within minutes. Run the full regression tag nightly on a schedule trigger, where the extra voice minutes and runtime do not hold up a developer. Cekura's Action handles both, because each trigger can pass its own tag.

Can GitHub Actions test LiveKit and Pipecat voice agents?

Yes. LiveKit's own CLI runs simulations in a GitHub Actions job, in text by default, and exits non-zero on a failed scenario. Cekura's Action tests LiveKit and Pipecat agents from version v1.1.0 through the livekit_v2 and pipecat_v2 execution modes, using the platform credentials saved on the Cekura agent. Those modes take scenario IDs, not tags.