Tool use & agents · %
τ²-bench
Sierra's customer-service agent benchmark: the agent follows a policy and uses tools while an LLM-simulated user talks to it (retail, airline, telecom, banking).
Active Contamination risk: Medium 100 results · 43 models Updated Oct 22, 2026 Newest result: 2026-08-03 Official leaderboard ↗
What it measures
pass^1 per domain (pass^k in the source measures reliability over k repeats).
Limitations
Many rows are submitted by the model developer (labelled 'vendor'); prompt addenda and user-simulator model differ.
Contamination risk: Medium
Tasks are public in the repo.
Leaderboard · Subset / setting: retail
23 models
| # | Model | Provider | Score | Rank | Setting | Source type | Date |
|---|---|---|---|---|---|---|---|
| 1 | Gemini 3.0 Pro | 85.3% | 2 | retail domain, pass^1; submitted by Sierra; vv0.1.3 | Official leaderboard | Nov 18, 2025 | |
| 2 | Qwen3.5 397B-A17BOW | Alibaba (Qwen) | 84.4% | 3 | retail domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1“Qwen3.5-397B-A17B” | Official leaderboard | Feb 27, 2026 |
| 3 | Nemotron-Orchestrator-8B | NVIDIA | 84.2% | 4 | retail domain, pass^1; submitted by NVIDIA; v0.1.3 | Vendor-reported | Sep 15, 2025 |
| 4 | RAFT-30B-A3B | Northeastern University HAI Lab | 82.5% | 5 | retail domain, pass^1; submitted by Northeastern University HAI Lab | Vendor-reported | Apr 30, 2026 |
| 5 | Claude Opus 4.1 | Anthropic | 82.4% | 6 | retail domain, pass^1; submitted by Anthropic; voriginal tau-bench | Vendor-reported | Aug 5, 2025 |
| 6 | GPT-5.2 | OpenAI | 81.6% | 7 | retail domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 7 | GPT-5 | OpenAI | 81.6% | 7 | retail domain, pass^1; submitted by Sierra; v0.1.3 | Official leaderboard | Aug 9, 2025 |
| 8 | Claude Opus 4 | Anthropic | 81.4% | 9 | retail domain, pass^1; submitted by Anthropic; voriginal tau-bench | Vendor-reported | May 22, 2025 |
| 9 | DeepSeek-V3.2OW | DeepSeek | 81.1% | 10 | retail domain, pass^1; submitted by DeepSeek; vv0.1.3 | Vendor-reported | Dec 1, 2025 |
| 10 | Claude Sonnet 4 | Anthropic | 80.5% | 11 | retail domain, pass^1; submitted by Anthropic; voriginal tau-bench | Vendor-reported | May 22, 2025 |
| 11 | Claude Opus 4.5 | Anthropic | 79.6% | 12 | retail domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 12 | Gemini 3 Flash | 76.8% | 14 | retail domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | Mar 2, 2026 | |
| 13 | Gemini 3 Pro | 75.9% | 15 | retail domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | Mar 2, 2026 | |
| 14 | Qwen3-Max-Thinking-Preview | Alibaba (Qwen) | 75.4% | 16 | retail domain, pass^1; submitted by Qwen; vv0.1.3 | Vendor-reported | Oct 30, 2025 |
| 15 | GPT-4.1 | OpenAI | 74.0% | 18 | retail domain, pass^1; submitted by Sierra; v0.1.3 | Official leaderboard | Jun 9, 2025 |
| 16 | o3 | OpenAI | 73.9% | 19 | retail domain, pass^1; submitted by OpenAI; v0.1.3 | Vendor-reported | Apr 16, 2025 |
| 17 | GLM-5OW | Zhipu AI (Z.ai) | 73.7% | 20 | retail domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1 | Official leaderboard | Feb 27, 2026 |
| 18 | Claude Sonnet 4.5 | Anthropic | 72.4% | 21 | retail domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1 | Official leaderboard | Feb 24, 2026 |
| 19 | Qwen3-Max | Alibaba (Qwen) | 72.2% | 22 | retail domain, pass^1; submitted by Alibaba; v0.1.3 | Official leaderboard | Sep 23, 2024 |
| 20 | Claude 3.7 Sonnet | Anthropic | 72.1% | 23 | retail domain, pass^1; submitted by Sierra; v0.1.3“Claude-3.7-Sonnet” | Official leaderboard | Jun 9, 2025 |
| 21 | Kimi K2 ThinkingOW | Moonshot AI | 70.6% | 24 | retail domain, pass^1; submitted by Moonshot AI; v0.1.3“Kimi-k2” | Vendor-reported | Jul 28, 2025 |
| 22 | o4-mini | OpenAI | 68.3% | 25 | retail domain, pass^1; submitted by Sierra; v0.1.3 | Official leaderboard | Jun 9, 2025 |
| 23 | GPT-4.1 mini | OpenAI | 61.4% | 26 | retail domain, pass^1; submitted by Sierra; v0.1.3“GPT-4.1-mini” | Official leaderboard | Jun 9, 2025 |
Leaderboard · Subset / setting: airline
22 models
| # | Model | Provider | Score | Rank | Setting | Source type | Date |
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.5 | Anthropic | 84.0% | 1 | airline domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 2 | GPT-5.2 | OpenAI | 83.0% | 2 | airline domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 3 | Gemini 3 Flash | 82.5% | 3 | airline domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | Mar 2, 2026 | |
| 4 | GLM-5OW | Zhipu AI (Z.ai) | 82.5% | 3 | airline domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1 | Official leaderboard | Feb 27, 2026 |
| 5 | Qwen3.5 397B-A17BOW | Alibaba (Qwen) | 81.5% | 5 | airline domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1“Qwen3.5-397B-A17B” | Official leaderboard | Feb 27, 2026 |
| 6 | Gemini 3 Pro | 80.5% | 6 | airline domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | Mar 2, 2026 | |
| 7 | Gemini 3.0 Pro | 73.0% | 7 | airline domain, pass^1; submitted by Sierra; vv0.1.3 | Official leaderboard | Nov 18, 2025 | |
| 8 | Claude Sonnet 4.5 | Anthropic | 72.0% | 8 | airline domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1 | Official leaderboard | Feb 24, 2026 |
| 9 | Qwen3-Max-Thinking-Preview | Alibaba (Qwen) | 71.0% | 9 | airline domain, pass^1; submitted by Qwen; vv0.1.3 | Vendor-reported | Oct 30, 2025 |
| 10 | Claude 3.7 Sonnet | Anthropic | 64.2% | 12 | airline domain, pass^1; submitted by Sierra; v0.1.3“Claude-3.7-Sonnet” | Official leaderboard | Jun 9, 2025 |
| 11 | DeepSeek-V3.2OW | DeepSeek | 63.8% | 13 | airline domain, pass^1; submitted by DeepSeek; vv0.1.3 | Vendor-reported | Dec 1, 2025 |
| 12 | GPT-5 | OpenAI | 62.5% | 14 | airline domain, pass^1; submitted by Sierra; v0.1.3 | Official leaderboard | Aug 9, 2025 |
| 13 | Claude Sonnet 4 | Anthropic | 60.0% | 15 | airline domain, pass^1; submitted by Anthropic; voriginal tau-bench | Vendor-reported | May 22, 2025 |
| 14 | Claude Opus 4 | Anthropic | 59.6% | 16 | airline domain, pass^1; submitted by Anthropic; voriginal tau-bench | Vendor-reported | May 22, 2025 |
| 15 | Qwen3-Max | Alibaba (Qwen) | 59.5% | 17 | airline domain, pass^1; submitted by Alibaba; v0.1.3 | Official leaderboard | Sep 23, 2024 |
| 16 | Kimi K2 ThinkingOW | Moonshot AI | 56.5% | 18 | airline domain, pass^1; submitted by Moonshot AI; v0.1.3“Kimi-k2” | Vendor-reported | Jul 28, 2025 |
| 17 | Claude Opus 4.1 | Anthropic | 56.0% | 19 | airline domain, pass^1; submitted by Anthropic; voriginal tau-bench | Vendor-reported | Aug 5, 2025 |
| 18 | GPT-4.1 | OpenAI | 56.0% | 19 | airline domain, pass^1; submitted by Sierra; v0.1.3 | Official leaderboard | Jun 9, 2025 |
| 19 | Nemotron-Orchestrator-8B | NVIDIA | 56.0% | 19 | airline domain, pass^1; submitted by NVIDIA; v0.1.3 | Vendor-reported | Sep 15, 2025 |
| 20 | o4-mini | OpenAI | 52.1% | 23 | airline domain, pass^1; submitted by Sierra; v0.1.3 | Official leaderboard | Jun 9, 2025 |
| 21 | o3 | OpenAI | 52.0% | 24 | airline domain, pass^1; submitted by OpenAI; v0.1.3 | Vendor-reported | Apr 16, 2025 |
| 22 | GPT-4.1 mini | OpenAI | 48.7% | 25 | airline domain, pass^1; submitted by Sierra; v0.1.3“GPT-4.1-mini” | Official leaderboard | Jun 9, 2025 |
Leaderboard · Subset / setting: banking
27 models
| # | Model | Provider | Score | Rank | Setting | Source type | Date |
|---|---|---|---|---|---|---|---|
| 1 | Qwen 3.8 Max | Alibaba (Qwen) | 55.2% | 1 | banking_knowledge domain, pass^1; effort: xhigh; submitted by Sierra; v1.0.1 | Official leaderboard | Aug 3, 2026 |
| 2 | Claude Opus 5 | Anthropic | 48.7% | 2 | banking_knowledge domain, pass^1; effort: max; submitted by Sierra; v1.0.1 | Official leaderboard | Aug 3, 2026 |
| 3 | Grok 4.5 | xAI | 47.9% | 3 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | Jul 24, 2026 |
| 4 | GPT-5.6 Sol | OpenAI | 46.9% | 4 | banking_knowledge domain, pass^1; effort: xhigh; submitted by Sierra; v1.0.1“GPT-5.6-sol” | Official leaderboard | Jul 22, 2026 |
| 5 | GPT-5.5 | OpenAI | 44.6% | 5 | banking_knowledge domain, pass^1; effort: xhigh; submitted by Sierra; v1.0.1 | Official leaderboard | Jul 22, 2026 |
| 6 | Muse Spark 1.1 | Meta | 40.5% | 6 | banking_knowledge domain, pass^1; effort: xhigh; submitted by Sierra; v1.0.1 | Official leaderboard | Jul 23, 2026 |
| 7 | Claude Opus 4.7 | Anthropic | 40.2% | 7 | banking_knowledge domain, pass^1; effort: max; submitted by Sierra; v1.0.1 | Official leaderboard | Jul 23, 2026 |
| 8 | Claude Fable 5 | Anthropic | 39.7% | 8 | banking_knowledge domain, pass^1; effort: max; submitted by Sierra; v1.0.1 | Official leaderboard | Jul 23, 2026 |
| 9 | Claude Opus 4.8 | Anthropic | 39.7% | 8 | banking_knowledge domain, pass^1; effort: max; submitted by Sierra; v1.0.1 | Official leaderboard | Jul 23, 2026 |
| 10 | GPT-5.4 | OpenAI | 39.4% | 10 | banking_knowledge domain, pass^1; effort: xhigh; submitted by Sierra; v1.0.1 | Official leaderboard | May 6, 2026 |
| 11 | GLM-5.2OW | Zhipu AI (Z.ai) | 37.1% | 11 | banking_knowledge domain, pass^1; effort: xhigh; submitted by Sierra; v1.0.1 | Official leaderboard | Jul 24, 2026 |
| 12 | Kimi K3OW | Moonshot AI | 37.1% | 11 | banking_knowledge domain, pass^1; effort: max; submitted by Sierra; v1.0.1 | Official leaderboard | Jul 24, 2026 |
| 13 | GPT-5.2 | OpenAI | 32.2% | 13 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 14 | Distyl ButtonAgent | Distyl AI | 31.2% | 14 | banking_knowledge domain, pass^1; effort: high; submitted by Distyl AI; v0.2.1-dev | Vendor-reported | Mar 25, 2026 |
| 15 | Claude Opus 4.6 | Anthropic | 27.3% | 15 | banking_knowledge domain, pass^1; effort: max; submitted by Sierra; v1.0.1 | Official leaderboard | May 6, 2026 |
| 16 | Gemini 3 Flash | 27.3% | 15 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | Mar 2, 2026 | |
| 17 | Gemini 3.1 Pro | 26.0% | 17 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1“Gemini 3.1 Pro Preview” | Official leaderboard | May 5, 2026 | |
| 18 | Claude Sonnet 4.5 | Anthropic | 25.3% | 18 | banking_knowledge domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1 | Official leaderboard | Feb 24, 2026 |
| 19 | InklingOW | Thinking Machines | 25.0% | 19 | banking_knowledge domain, pass^1; effort: max; submitted by Sierra; v1.0.1 | Official leaderboard | Jul 24, 2026 |
| 20 | Claude Opus 4.5 | Anthropic | 24.7% | 20 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 21 | Gemini 3 Pro | 18.0% | 21 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | Mar 2, 2026 | |
| 22 | Grok 4.2 | xAI | 18.0% | 21 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 23 | Grok 4 fast | xAI | 15.7% | 23 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 24 | Gemini 2.5 Pro (Jun 2025) | 13.7% | 24 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1“Gemini 2.5 Pro” | Official leaderboard | May 5, 2026 | |
| 25 | Grok 4.1 fast | xAI | 13.1% | 25 | banking_knowledge domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 26 | GLM-5OW | Zhipu AI (Z.ai) | 9.8% | 27 | banking_knowledge domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1 | Official leaderboard | Feb 27, 2026 |
| 27 | Qwen3.5 397B-A17BOW | Alibaba (Qwen) | 9.8% | 27 | banking_knowledge domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1“Qwen3.5-397B-A17B” | Official leaderboard | Feb 27, 2026 |
Leaderboard · Subset / setting: telecom
18 models
| # | Model | Provider | Score | Rank | Setting | Source type | Date |
|---|---|---|---|---|---|---|---|
| 1 | Gemini 3.0 Pro | 98.0% | 2 | telecom domain, pass^1; submitted by Sierra; vv0.1.3 | Official leaderboard | Nov 18, 2025 | |
| 2 | Qwen3.5 397B-A17BOW | Alibaba (Qwen) | 97.8% | 4 | telecom domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1“Qwen3.5-397B-A17B” | Official leaderboard | Feb 27, 2026 |
| 3 | DeepSeek-V3.2OW | DeepSeek | 96.2% | 5 | telecom domain, pass^1; submitted by DeepSeek; vv0.1.3 | Vendor-reported | Dec 1, 2025 |
| 4 | GPT-5 | OpenAI | 95.8% | 6 | telecom domain, pass^1; submitted by Sierra; v0.1.3 | Official leaderboard | Aug 9, 2025 |
| 5 | Qwen3-Max-Thinking-Preview | Alibaba (Qwen) | 95.8% | 6 | telecom domain, pass^1; submitted by Qwen; vv0.1.3 | Vendor-reported | Oct 30, 2025 |
| 6 | Claude Opus 4.5 | Anthropic | 92.3% | 8 | telecom domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 7 | Gemini 3 Flash | 91.2% | 9 | telecom domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | Mar 2, 2026 | |
| 8 | Gemini 3 Pro | 91.0% | 10 | telecom domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | Mar 2, 2026 | |
| 9 | GPT-5.2 | OpenAI | 89.7% | 11 | telecom domain, pass^1; effort: high; submitted by Sierra; v1.0.1 | Official leaderboard | May 5, 2026 |
| 10 | Nemotron-Orchestrator-8B | NVIDIA | 88.6% | 12 | telecom domain, pass^1; submitted by NVIDIA; v0.1.3 | Vendor-reported | Sep 15, 2025 |
| 11 | GLM-5OW | Zhipu AI (Z.ai) | 86.8% | 13 | telecom domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1 | Official leaderboard | Feb 27, 2026 |
| 12 | Claude Sonnet 4.5 | Anthropic | 84.9% | 14 | telecom domain, pass^1; effort: enabled; submitted by Sierra; v1.0.1 | Official leaderboard | Feb 24, 2026 |
| 13 | Qwen3-Max | Alibaba (Qwen) | 84.2% | 15 | telecom domain, pass^1; submitted by Alibaba; v0.1.3 | Official leaderboard | Sep 23, 2024 |
| 14 | Kimi K2 ThinkingOW | Moonshot AI | 65.8% | 16 | telecom domain, pass^1; submitted by Moonshot AI; v0.1.3“Kimi-k2” | Vendor-reported | Jul 28, 2025 |
| 15 | o4-mini | OpenAI | 50.2% | 18 | telecom domain, pass^1; submitted by Sierra; v0.1.3 | Official leaderboard | Jun 9, 2025 |
| 16 | Claude 3.7 Sonnet | Anthropic | 49.0% | 19 | telecom domain, pass^1; submitted by Sierra; v0.1.3“Claude-3.7-Sonnet” | Official leaderboard | Jun 9, 2025 |
| 17 | GPT-4.1 mini | OpenAI | 48.9% | 20 | telecom domain, pass^1; submitted by Sierra; v0.1.3“GPT-4.1-mini” | Official leaderboard | Jun 9, 2025 |
| 18 | GPT-4.1 | OpenAI | 34.0% | 21 | telecom domain, pass^1; submitted by Sierra; v0.1.3 | Official leaderboard | Jun 9, 2025 |
Best score over time
Each step is a new best result in our data, by evaluation date.
- 2024-09-2372.2%Qwen3-Max
- 2025-06-0974.0%GPT-4.1
- 2025-08-0981.6%GPT-5
- 2025-11-1885.3%Gemini 3.0 Pro
Data sources & licences
| Source | Licence / terms | How we reuse it | Last fetched |
|---|---|---|---|
| τ²-bench leaderboard submissions Sierra Research, τ²-bench | MIT (repo) | Open licence – full data with attribution | Oct 22, 2026 |
FAQ
What does τ²-bench measure?
Sierra's customer-service agent benchmark: the agent follows a policy and uses tools while an LLM-simulated user talks to it (retail, airline, telecom, banking). pass^1 per domain (pass^k in the source measures reliability over k repeats).
How reliable is τ²-bench?
Many rows are submitted by the model developer (labelled 'vendor'); prompt addenda and user-simulator model differ. Contamination risk: Medium – Tasks are public in the repo.