Tau-Bench Explained: Tau2, Tau3, and Pass^k (2026)

Tau-Bench is Sierra's benchmark for customer service agents. An LLM plays the customer, and the agent calls API tools under a written policy. A grader checks the final database state.

A model can handle a task once and miss it on the eighth try, so the benchmark scores repeated trials with pass^k.

I read the papers and the tau2-bench repo, then opened the raw leaderboard files to map the 4 versions and 5 domains with their scores, and to flag where the numbers stop being comparable.

After that reading, my short take is that tau-bench ranks models on policy-bound tool use, and your deployed voice agent needs its own test on the platform it's hosted on.

TL;DR: tau-bench at a glance

• Tau-bench is Sierra's customer service agent benchmark: first released in June 2024, with 115 retail and 50 airline tasks in the original release.

• It has 4 major generations: tau-bench (retail and airline), tau2-bench (telecom), tau3-bench (banking and voice), and 2.0 (October 19, 2026, 6 languages).

• Pass^k counts a task only if every trial succeeds: in the original paper, GPT-4o's retail score fell from 61.2% at pass^1 to under 25% at pass^8.

What is tau-bench?

Tau-bench is a benchmark that tests whether an AI agent can finish a customer service task by talking to a simulated user and calling tools. Sierra published it in June 2024 in a paper by Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.

Each task gives the agent a database of realistic records, API tools to read and change it, and a policy document it has to follow. An LLM plays the customer from a hidden scenario written for that task.

Grading looks at the end state. The grader compares the database after the conversation with an annotated goal state. The paper then adds a second check to verify that the agent provided the required information to the user.

The original release had 115 retail tasks and 50 airline tasks. In Sierra's launch post, the best of 12 models tested, a GPT-4o function-calling agent, averaged under 50% success across the 2 domains.

Tau-bench versions: tau2-bench, tau3 bench, and 2.0

The tau benchmark family has had 4 releases since 2024, and scores from different releases are not comparable. The GitHub repo is still named tau2-bench, even though its README now covers tau3 bench and 2.0.

tau-bench (June 2024)

Sierra posted the paper on arXiv on June 17, 2024, and the launch post on June 20. It covers retail and airline only. The README of the original repo warns that those tasks are no longer updated and points readers to the newer code.

tau2-bench (June 2025)

Tau2-bench adds dual control, which lets the user act on the world too. In the new telecom domain, the agent has to talk a user through steps only the user can perform on their own device, such as rebooting it or changing a network setting.

The paper went up on June 9, 2025. Sierra's launch post reports task success falling by up to 25 points when an agent moves from acting alone to guiding a user.

tau3 bench (February to March 2026)

Tau3 bench bundles 3 additions released in February and March 2026. Sierra's March 18 post summarizes them:

• A task audit: taubench.com lists 50+ airline and retail tasks fixed for wrong expected actions, ambiguous instructions, and impossible constraints. The v1.0.0 release notes count 75+ fixes across domains.

• Tau-knowledge: a banking domain with 97 tasks over 698 policy documents. A March 4 paper put the best frontier result at about 25.5% pass^1.

• Tau-voice: full-duplex voice on 278 tasks. A March 14 paper reports 31 to 51% task success in clean audio and 26 to 38% with noise and accents.

tau-bench 2.0 (October 2026)

Tau-bench 2.0 came out on October 19, 2026. Its GitHub release introduces tau-i18n (called tau-multilingual before the release): the existing tasks from 3 domains in 6 languages (English, Spanish, Portuguese, Hindi, Chinese, and Korean), in text and voice.

It also includes tau-entity (the intake domain), 200 tasks on capturing values such as names and addresses exactly on an outbound call, plus a post-hoc judge suite that never changes the reward.

The companion paper covers 4,500 full-duplex calls. Korean scored 14.7 points below English on task completion, and Mandarin 8.4 points below, while the other 3 languages stayed within 3.2 points.

Sierra’s README says 2.0 scores can’t be compared with the 1.x leaderboard and that the two appear on separate leaderboard tabs.

How does tau-bench work?

A tau-bench run moves through 4 steps:

1. The agent gets its setup: a policy document and API tools for one domain, plus the database behind them.

2. A simulated user opens the conversation: an LLM follows a hidden scenario and talks to the agent turn by turn. In the telecom domain, the user also holds tools of its own.

3. The agent works the task: it asks questions and checks policy before it uses tools that read or write records. In voice mode, both sides speak at once (full-duplex), and the caller can interrupt or wait.

4. The grader compares end states: the final database has to match the goal state.

Before 2.0, a second check looked for required values as substring matches in the agent’s replies. Since 2.0, an LLM judge scores those communication checks as part of the reward, separate from the post-hoc judge suite.

Because the user is simulated, a re-run keeps the task but changes the conversation, and that is what gives pass^k its repeated trials.

What is pass^k in tau-bench?

Pass^k is the probability that an agent succeeds on all k trials of the same task, averaged across tasks. For a task run n times with c successes, the paper's estimate is C(c,k) divided by C(n,k).

Suppose a task passes in 6 of 8 trials (made-up numbers). That's 75% at pass^1 and 0% at pass^8.

In the paper, GPT-4o with function calling scored 61.2% on retail at pass^1 and under 25% at pass^8. Sierra’s 2024 launch post describes that as a 60% drop relative to pass^1 and reads it as a 25% chance of resolving 8 cases of one issue for different customers.

The pattern holds on current boards. On the banking board, Claude Opus 5 scores 48.7% at pass^1 and 32.0% at pass^4 over 4 trials, according to its submission file.

The multilingual voice board can’t show this drop. Its entries use 1 trial per task, so it doesn't report pass^k.

Pass@k, by contrast, asks whether at least one of k attempts worked. Cekura Bench borrows pass^k rather than pass@k. Its pass^3 counts a scenario only when all 3 runs pass, a definition the benchmark write-up traces to the tau-bench paper.

Tau-bench domains

Tau-bench has 5 domains today: retail, airline, telecom, banking, and intake.

Airline tasks include changing a flight reservation. Telecom tasks cover broken data connections and MMS problems. Banking tasks include opening and closing accounts, redeeming referral promotions, and handling direct-deposit inquiries.

The multilingual voice board scores the first 3 domains in the table. Banking and intake have separate voice boards. Telecom is where I'd start if dual control matters to you, because the simulated user there can change settings on the device too.

Tau-bench leaderboard scores

These are the top 3 entries on the text, banking, and multilingual voice boards at taubench.com as of November 2, 2026. I read the submission files behind the site and recomputed the averages.

The table comes with 3 caveats:

• Older entries are behind a toggle. Gemini 3 Pro’s 85.4% would place second, but it ran on tau2-bench v0.1.3 with Gemini 3.0 Pro as the user simulator, so taubench.com shows it only under “Legacy (v1)”. The default top 3 all used v1.0.1.

• Banking grading changed in v1.0.1. The release notes show re-graded pass^1 rising by up to about 9 points; GPT-5.5 went from 37.4% to 46.4%. So the 25.5% from March 2026 is a pre-fix number.

• The voice board is 2.0 data. It covers 6 languages and 1 trial per task, with a GPT-5.5 caller simulator. Ava Zero's 86.0% is a platform result, because the entry runs on a hosted agent platform.

Tau-bench pros and cons

Tau-bench grades the final database state and repeats each task across trials, and most of its strengths come from those two choices. Its limits come from how often the versions change and from the simulated user.

Pros

• Outcome grading: grading depends on the final database. A wrong refund shows up even when the transcript sounds friendly.

• Pass^k shows inconsistency: it shows Claude Opus 5 falling from 48.7% at pass^1 to 32.0% at pass^4 on banking.

• Open code and data: the repo is public, and the 2.0 task sets don't change. All 29 current (non-legacy) text submissions on taubench.com include their trajectories.

• Text and voice coverage: tau-bench covers both, with 6 languages across 3 domains in 2.0.

Cons

• Scores don't carry across versions: 2.0 starts a new leaderboard, and banking grading changed in v1.0.1.

• The user is an LLM: 2.0 changed the default simulator from GPT-4.1 to GPT-5.5, so scores from the two versions used different simulators.

• The scope is customer service: the 5 domains include no coding or research tasks.

Who should use tau-bench

Tau-bench is for teams choosing models for tool-calling agents bound by a written policy.

Use it if you:

• Compare LLMs for support-style agents that read and write records

• Want a public benchmark with a fixed task set, whose submissions and trajectories anyone can inspect

• Build on realtime voice APIs and want to measure how much a model loses when it moves from text to voice (the tau-voice paper found voice agents kept 30 to 45% of their text-mode capability)

Skip it if you:

• Ship an agent on a voice platform and need to test the whole call path

• Need your own policies and workflows in the test set, since you'd have to author a new domain yourself

How to run tau-bench in 5 steps

Running tau-bench costs API credits for the agent and user models, plus any hosting you pay for. The code is MIT-licensed and free to use.

1. Install uv and a supported Python. Tau-bench needs Python 3.12 or newer, below 3.14.

2. Clone the repo and sync dependencies:

git clone https://github.com/sierra-research/tau2-bench
cd tau2-bench
uv sync    # add --extra voice for voice runs

3. Add your API keys: copy .env.example to .env and set OPENAI_API_KEY. Tau-bench calls models through LiteLLM, and any supported provider works.

4. Print the plan with a dry run. The full text-v1 benchmark is 900 calls at 1 trial. Read the plan before you launch it:

tau2 run --benchmark text --agent-llm openai/gpt-5.5 \
  --agent-llm-args '{"reasoning_effort":"low"}' --save-to text-run --dry-run

5. Run a quick 5-task test:

tau2 run --domain airline --agent-llm gpt-4.1 --user-llm gpt-4.1 \
  --num-trials 1 --num-tasks 5

Tau-bench writes results to data/simulations/, and tau2 view opens them.

Pro tip: for scores comparable with the 1.x leaderboard, use the v1.0.1 tag.

Voice also needs --extra voice with portaudio and ffmpeg installed, plus your own caller voices, since the shipped voice IDs are Sierra-internal. Cascaded adapters such as LiveKit and Deepgram were listed as experimental in the v1.0.0 notes.

What tau-bench does not measure

A tau-bench score measures a model in a simulated conversation, and your production system isn't part of that test. That leaves 4 limits, plus a caveat about who runs the benchmark:

• A reward of 1 can hide a policy violation. The paper calls a reward of 1 necessary but not sufficient since an agent might issue a return without the user’s explicit confirmation, which breaks policy, and still be rewarded.

• Voice entries use realtime model APIs. 3 of the 5 entries on the multilingual voice board are realtime models from OpenAI, xAI, and Google, so each score reflects the model alone.

Ava Zero is one of the two exceptions. Ava Zero connected through a Pipecat adapter to a hosted agent platform, and the submission notes name GLM 5.3 as the front end and DeepSeek v4.1 Flash as the backend worker.

• The caller sets its own timing. The LLM decides when to interrupt, and the timing changes from one trial to the next. In a review of 91 failed calls, Sierra’s tau-voice paper attributes 79 to 90% of failures to agent behavior, so the simulator explains the smaller share, though it still keeps trials from repeating exactly.

• The tasks are fixed. The benchmark ships with its own refund policy and tool schemas, so your policies and tools get no coverage.

• Both sources have a stake. Sierra builds customer service agents, and Cekura's comparison post frames that as an incentive. Cekura sells testing and publishes a benchmark, so treat both views with equal caution.

Cekura’s post also argues that testing models via their APIs leaves your platform, prompt, and tools out of the score.

Cekura Bench responds by putting an LLM-driven caller inside a rule-based framework with interruptions at exact moments. It tests deployed agents on Vapi, Retell, ElevenLabs, LiveKit, Pipecat, and Telnyx, with 82 scenarios, each repeated 3 times.

My verdict on tau-bench

Use tau-bench to compare models on policy-bound tool use, and read it at pass^k. Pin the version and record the user simulator, then run at least 4 trials, as the Claude Opus 5 banking submission does. To build evals around your own agent, the guide to AI agent evals describes an 8-step process.

For deployed voice agents, test the call itself. Cekura runs simulated calls against the agent you ship:

• Pre-production: conditional actions trigger an interruption or a fault at an exact moment, and repeat runs show whether a pass holds up.

• Infrastructure: interruptions, background noise, latency, and WebRTC performance.

• Observability: production monitoring runs metrics on live calls.

Cekura has native integrations for the 6 platforms Cekura Bench covers, plus Bland.

Cekura supports SOC 2, HIPAA, and GDPR compliance, and the platform covers transcript redaction and role-based access with audit trails.

Cekura's own board compares platforms rather than models. On the voice platform board (9 configurations, last updated October 22, 2026), PolyAI leads at 87.8% pass^3.

That lead has limits because providers chose their own configurations, which Cekura then measured under its published method, and PolyAI’s entry had no scored coverage for the repetition or call-termination metrics.

Book a demo to see Cekura run tau-bench-style repeat trials, with simulated interruptions, against the agent you ship.

Frequently asked questions

What is tau-bench?

Tau-bench is a benchmark from Sierra that tests AI agents on customer service tasks with an LLM-simulated user and a written policy. It grades each run by comparing the final database to a goal state, and it reports consistency with pass^k.

What is tau2-bench?

Tau2-bench is the June 2025 release that added dual control, where the user can act on the environment too, and a telecom domain. The GitHub repo is still named tau2-bench and now holds the tau3 bench and 2.0 code.

What is tau3 bench?

Tau3 bench is the early-2026 expansion that added a banking domain (97 tasks over 698 documents) and full-duplex voice, and it fixed errors in a round of existing tasks. Sierra's v1.0 release carries that name.

What does pass^k mean in tau-bench?

Pass^k is the probability that an agent solves the same task on all k trials, averaged across tasks. A higher k is a stricter test, and pass^1 is the single-trial success rate.

Is tau-bench open source?

Yes. The tau2-bench repo is MIT-licensed, and Sierra accepts leaderboard submissions through a guide in the repo, linked from taubench.com.

Are AI agent benchmarks like tau-bench trustworthy?

Yes, as a measure of model behavior. Tau-bench scores a model against a simulated user in fixed domains, and a high score shows the model can follow policy on those tasks. The platform and prompt your agent ships with need their own tests.

Before comparing scores, confirm the version and the simulator. Then count how many trials each run used.

What is the best tool for testing AI agents on live calls?

Cekura is one option built for testing AI agents on live calls. It simulates calls against the agent you ship, with interruptions and repeat runs, while tau-bench compares underlying models.