Agentic coding · %
SWE-bench Verified
500 human-validated GitHub issues from 12 Python repositories; the system must produce a patch that makes the hidden tests pass.
Contested Contamination risk: High 142 results · 81 models Updated Oct 22, 2026 Newest result: 2026-06-25 Official leaderboard ↗
What it measures
% of issues resolved by a model + agent scaffold. Rows from swebench.com are agent systems; rows from Epoch AI use Epoch's own scaffold.
Limitations
In February 2026 OpenAI stopped reporting SWE-bench Verified, saying it is contaminated and that 59.4% of 138 audited hard tasks had flawed tests or descriptions. Scores depend heavily on the agent scaffold.
Contamination risk: High
OpenAI found frontier models could reproduce gold patches from memory.
Leaderboard · Subset / setting: epoch
32 models
| # | Model | Provider | Score | Rank | Setting | Source type | Date |
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | Anthropic | 83.5% | 1 | Epoch AI run (Epoch scaffold); effort: max | Independent run | Apr 20, 2026 |
| 2 | GPT-5.5 | OpenAI | 80.6% | 2 | Epoch AI run (Epoch scaffold); effort: xhigh | Independent run | Apr 24, 2026 |
| 3 | Gemini 3.5 Flash | 79.3% | 3 | Epoch AI run (Epoch scaffold) | Independent run | Jun 1, 2026 | |
| 4 | Claude Opus 4.6 | Anthropic | 78.7% | 4 | Epoch AI run (Epoch scaffold); effort: no thinking | Independent run | Feb 18, 2026 |
| 5 | GLM-5.2OW | Zhipu AI (Z.ai) | 78.7% | 5 | Epoch AI run (Epoch scaffold) | Independent run | Jun 25, 2026 |
| 6 | DeepSeek-V4-ProOW | DeepSeek | 77.6% | 6 | Epoch AI run (Epoch scaffold); effort: max | Independent run | Jun 18, 2026 |
| 7 | Qwen3.7-Max | Alibaba (Qwen) | 77.3% | 7 | Epoch AI run (Epoch scaffold) | Independent run | Jun 18, 2026 |
| 8 | GPT-5.4 | OpenAI | 76.9% | 8 | Epoch AI run (Epoch scaffold); effort: high | Independent run | Mar 6, 2026 |
| 9 | Qwen 3.6 Max (Preview) | Alibaba (Qwen) | 76.7% | 9 | Epoch AI run (Epoch scaffold); effort: Preview | Independent run | May 28, 2026 |
| 10 | Kimi K2.6OW | Moonshot AI | 76.7% | 9 | Epoch AI run (Epoch scaffold) | Independent run | May 8, 2026 |
| 11 | Claude Opus 4.5 | Anthropic | 76.7% | 9 | Epoch AI run (Epoch scaffold); effort: no thinking | Independent run | Feb 5, 2026 |
| 12 | Gemini 3.1 Pro | 75.6% | 12 | Epoch AI run (Epoch scaffold) | Independent run | Feb 24, 2026 | |
| 13 | Gemini 3 Flash | 75.4% | 13 | Epoch AI run (Epoch scaffold) | Independent run | Feb 18, 2026 | |
| 14 | Claude Sonnet 4.6 | Anthropic | 75.2% | 14 | Epoch AI run (Epoch scaffold); effort: no thinking | Independent run | Feb 21, 2026 |
| 15 | GPT-5.3 Codex | OpenAI | 74.8% | 15 | Epoch AI run (Epoch scaffold); effort: high | Independent run | Feb 25, 2026 |
| 16 | GLM-5.1OW | Zhipu AI (Z.ai) | 74.2% | 16 | Epoch AI run (Epoch scaffold) | Independent run | May 15, 2026 |
| 17 | Kimi K2.5OW | Moonshot AI | 73.8% | 17 | Epoch AI run (Epoch scaffold) | Independent run | Feb 17, 2026 |
| 18 | GPT-5.2 | OpenAI | 73.8% | 17 | Epoch AI run (Epoch scaffold); effort: high | Independent run | Feb 12, 2026 |
| 19 | GPT-5 | OpenAI | 73.5% | 19 | Epoch AI run (Epoch scaffold); effort: high | Independent run | Feb 6, 2026 |
| 20 | Claude Opus 4.1 | Anthropic | 73.3% | 20 | Epoch AI run (Epoch scaffold) | Independent run | Feb 11, 2026 |
| 21 | Gemini 3 Pro | 72.9% | 21 | Epoch AI run (Epoch scaffold) | Independent run | Feb 13, 2026 | |
| 22 | GLM-5OW | Zhipu AI (Z.ai) | 72.1% | 22 | Epoch AI run (Epoch scaffold) | Independent run | Feb 15, 2026 |
| 23 | Claude Sonnet 4.5 | Anthropic | 71.3% | 24 | Epoch AI run (Epoch scaffold); effort: no thinking | Independent run | Feb 5, 2026 |
| 24 | Claude Opus 4 | Anthropic | 70.7% | 25 | Epoch AI run (Epoch scaffold) | Independent run | Feb 6, 2026 |
| 25 | GPT-5.1 | OpenAI | 68.0% | 26 | Epoch AI run (Epoch scaffold); effort: high | Independent run | Feb 18, 2026 |
| 26 | GPT-5 mini | OpenAI | 64.7% | 27 | Epoch AI run (Epoch scaffold); effort: medium | Independent run | Feb 1, 2026 |
| 27 | o3 | OpenAI | 62.3% | 28 | Epoch AI run (Epoch scaffold); effort: medium | Independent run | Feb 12, 2026 |
| 28 | Claude 3.7 Sonnet | Anthropic | 61.0% | 29 | Epoch AI run (Epoch scaffold) | Independent run | Feb 4, 2026 |
| 29 | Qwen 3.6 Plus | Alibaba (Qwen) | 57.9% | 30 | Epoch AI run (Epoch scaffold); effort: 2026-04-02 | Independent run | May 14, 2026 |
| 30 | Gemini 2.5 Pro (Jun 2025) | 57.6% | 31 | Epoch AI run (Epoch scaffold) | Independent run | Feb 13, 2026 | |
| 31 | GPT-4.1 | OpenAI | 48.5% | 32 | Epoch AI run (Epoch scaffold) | Independent run | Feb 8, 2026 |
| 32 | GPT-4o (Nov 2024) | OpenAI | 31.0% | 33 | Epoch AI run (Epoch scaffold); effort: Nov 2024 | Independent run | Feb 11, 2026 |
Leaderboard · Subset / setting: leaderboard
64 models
| # | Model | Provider | Score | Rank | Setting | Source type | Date |
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.5 | Anthropic | 79.2% | 1 | agent: Sonar Foundation Agent“claude-opus-4-5” | Official leaderboard | Dec 5, 2025 |
| 2 | Gemini 3 Pro | 77.4% | 3 | agent: live-SWE-agent“gemini-3-pro-preview” | Official leaderboard | Nov 20, 2025 | |
| 3 | Claude Sonnet 4 | Anthropic | 76.8% | 4 | agent: EPAM AI/Run Developer Agent“claude-sonnet-4-20250514” | Official leaderboard | Aug 4, 2025 |
| 4 | claude-4-5-opus | Anthropic | 76.8% | 4 | agent: mini-SWE-agent | Official leaderboard | Feb 17, 2026 |
| 5 | Gemini 3 Flash | 75.8% | 6 | agent: mini-SWE-agent“gemini-3-flash-preview” | Official leaderboard | Feb 17, 2026 | |
| 6 | MiniMax-M2.5OW | MiniMax | 75.8% | 6 | agent: mini-SWE-agent“minimax-m2.5” | Official leaderboard | Feb 17, 2026 |
| 7 | Claude Opus 4.6 | Anthropic | 75.6% | 8 | agent: mini-SWE-agent“claude-opus-4-6” | Official leaderboard | Feb 17, 2026 |
| 8 | Claude Sonnet 4.5 | Anthropic | 74.8% | 9 | agent: Sonar Foundation Agent“claude-sonnet-4-5” | Official leaderboard | Nov 3, 2025 |
| 9 | claude-4-sonnet | Anthropic | 74.6% | 11 | agent: Lingxi-v1.5“claude-4-sonnet-20250514” | Official leaderboard | Jul 20, 2025 |
| 10 | GPT-5 | OpenAI | 74.4% | 12 | agent: Prometheus-v1.2.1“gpt-5-2025-08-07” | Official leaderboard | Oct 15, 2025 |
| 11 | claude-4-opus | Anthropic | 73.2% | 15 | agent: Tools“claude-4-opus-20250514” | Official leaderboard | May 22, 2025 |
| 12 | GLM-5OW | Zhipu AI (Z.ai) | 72.8% | 16 | agent: mini-SWE-agent“glm-5” | Official leaderboard | Feb 17, 2026 |
| 13 | GPT-5.2 | OpenAI | 72.8% | 16 | agent: mini-SWE-agent“gpt-5-2” | Official leaderboard | Feb 17, 2026 |
| 14 | GPT-5.2 Codex | OpenAI | 72.8% | 16 | agent: mini-SWE-agent“gpt-5-2-codex” | Official leaderboard | Feb 19, 2026 |
| 15 | Kimi K2 ThinkingOW | Moonshot AI | 71.2% | 22 | agent: Lingxi v1.5“kimi-k2-0905-preview” | Official leaderboard | Oct 14, 2025 |
| 16 | Kimi K2.5OW | Moonshot AI | 70.8% | 25 | agent: mini-SWE-agent“kimi-k2.5” | Official leaderboard | Feb 17, 2026 |
| 17 | DeepSeek-V3.2OW | DeepSeek | 70.0% | 28 | agent: mini-SWE-agent“deepseek-v3.2” | Official leaderboard | Feb 17, 2026 |
| 18 | Qwen3-Coder-480B-A35BOW | Alibaba (Qwen) | 69.6% | 29 | agent: OpenHands“https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct” | Official leaderboard | Aug 5, 2025 |
| 19 | GLM-4.6OW | Zhipu AI (Z.ai) | 68.2% | 30 | agent: Undisclosed“https://huggingface.co/zai-org/GLM-4.6” | Official leaderboard | Sep 30, 2025 |
| 20 | Claude Haiku 4.5 | Anthropic | 66.6% | 32 | agent: mini-SWE-agent“claude-haiku-4-5-20251001” | Official leaderboard | Feb 17, 2026 |
| 21 | Claude 3.7 Sonnet | Anthropic | 66.4% | 34 | agent: Aime-coder v1“claude-3-7-sonnet-20250219” | Official leaderboard | May 14, 2025 |
| 22 | GPT-5.1 | OpenAI | 66.0% | 35 | agent: mini-SWE-agent“gpt-5.1-2025-11-13” | Official leaderboard | Nov 20, 2025 |
| 23 | GPT-5.1-Codex | OpenAI | 66.0% | 35 | agent: mini-SWE-agent“gpt-5.1-codex” | Official leaderboard | Nov 24, 2025 |
| 24 | Kimi K2 (Jul 2025)OW | Moonshot AI | 65.4% | 37 | agent: OpenHands“moonshot/kimi-k2-0711-preview” | Official leaderboard | Jul 16, 2025 |
| 25 | o1-preview | OpenAI | 64.6% | 40 | agent: W&B Programmer O1 crosscheck5 | Official leaderboard | Jan 17, 2025 |
| 26 | o4-mini | OpenAI | 64.6% | 40 | agent: PatchPilot-v1.1 | Official leaderboard | May 3, 2025 |
| 27 | GLM-4.5OW | Zhipu AI (Z.ai) | 64.2% | 42 | agent: Undisclosed“https://huggingface.co/zai-org/GLM-4.5” | Official leaderboard | Jul 28, 2025 |
| 28 | Claude 3.5 Sonnet | Anthropic | 63.4% | 43 | agent: AgentScope“claude-3-5-sonnet-20241022” | Official leaderboard | Feb 6, 2025 |
| 29 | MiniMax-M2OW | MiniMax | 61.0% | 48 | agent: mini-SWE-agent“minimax-m2” | Official leaderboard | Nov 24, 2025 |
| 30 | Qwen3-Coder-30B-A3B-Instruct | Alibaba (Qwen) | 60.4% | 49 | agent: EntroPO + R2E | Official leaderboard | Sep 1, 2025 |
| 31 | deepseek-v3.2-reasoner | DeepSeek | 60.0% | 51 | agent: mini-SWE-agent | Official leaderboard | Dec 1, 2025 |
| 32 | GPT-5 mini | OpenAI | 59.8% | 52 | agent: mini-SWE-agent“gpt-5-mini-2025-08-07” | Official leaderboard | Aug 7, 2025 |
| 33 | /huggingface.co/agentica-org/DeepSWE-Preview | OpenAI | 58.8% | 53 | agent: DeepSWE-Preview“https://huggingface.co/agentica-org/DeepSWE-Preview” | Official leaderboard | Jun 29, 2025 |
| 34 | o3 | OpenAI | 58.4% | 54 | agent: mini-SWE-agent“o3-20250416” | Official leaderboard | Jul 26, 2025 |
| 35 | Devstral Small 2OW | Mistral AI | 56.4% | 56 | agent: mini-SWE-agent“devstral-small-2512” | Official leaderboard | Dec 9, 2025 |
| 36 | devstral-2512 | Mistral AI | 53.8% | 61 | agent: mini-SWE-agent | Official leaderboard | Dec 9, 2025 |
| 37 | FrogBoss-32B-2510 | — | 53.6% | 62 | agent: FrogBoss-32B-2510 | Official leaderboard | Nov 10, 2025 |
| 38 | Gemini 2.5 Pro (Jun 2025) | 53.6% | 62 | agent: mini-SWE-agent“gemini-2.5-pro” | Official leaderboard | Jul 26, 2025 | |
| 39 | /huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct | Alibaba (Qwen) | 51.6% | 67 | agent: OpenHands“https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct” | Official leaderboard | Aug 5, 2025 |
| 40 | Qwen2.5-Coder-32B-InstructOW | Alibaba (Qwen) | 47.0% | 70 | agent: Skywork-SWE-32B | Official leaderboard | Jun 16, 2025 |
| 41 | Devstral-Small-2505 | Mistral AI | 46.8% | 71 | agent: OpenHands“mistralai/Devstral-Small-2505” | Official leaderboard | May 20, 2025 |
| 42 | FrogMini-14B-2510 | — | 45.0% | 73 | agent: FrogMini-14B-2510 | Official leaderboard | Nov 10, 2025 |
| 43 | Gemini 2.0 Flash (Dec 2024) | 44.2% | 75 | agent: CodeShellAgent“gemini-2.0-flash-exp” | Official leaderboard | Jan 18, 2025 | |
| 44 | o3-mini | OpenAI | 42.4% | 77 | agent: Agentless Lite | Official leaderboard | Feb 14, 2025 |
| 45 | amazon.nova-premier-v1:0 | Amazon | 42.4% | 77 | agent: Amazon Nova Premier 1.0 | Official leaderboard | May 27, 2025 |
| 46 | DeepSeek-V3 (Mar 2025)OW | DeepSeek | 42.0% | 80 | agent: SWE-Exp“DeepSeek-V3-0324” | Official leaderboard | Aug 6, 2025 |
| 47 | Llama3-SWE-RL-70B | Meta | 41.2% | 81 | agent: Agentless Mini | Official leaderboard | Feb 26, 2025 |
| 48 | Claude 3 Haiku | Anthropic | 40.6% | 82 | agent: Tools“claude-3-haiku-20240307” | Official leaderboard | Oct 22, 2024 |
| 49 | Qwen 2.5 | Alibaba (Qwen) | 40.2% | 83 | agent: SWE-agent | Official leaderboard | May 11, 2025 |
| 50 | GPT-4.1 | OpenAI | 39.6% | 84 | agent: mini-SWE-agent“gpt-4.1-20250414” | Official leaderboard | Jul 26, 2025 |
| 51 | GPT-4o (May 2024) | OpenAI | 38.8% | 85 | agent: Agentless-1.5“gpt-4o-2024-05-13” | Official leaderboard | Oct 28, 2024 |
| 52 | devstral-small-2507 | Mistral AI | 38.0% | 87 | agent: SWE-agent | Official leaderboard | Jul 25, 2025 |
| 53 | GPT-5 nano | OpenAI | 34.8% | 88 | agent: mini-SWE-agent“gpt-5-nano-2025-08-07” | Official leaderboard | Aug 7, 2025 |
| 54 | Gemini 2.5 Flash (Jun 2025) | 28.7% | 92 | agent: mini-SWE-agent“gemini-2.5-flash” | Official leaderboard | Jul 26, 2025 | |
| 55 | gpt-oss-120bOW | OpenAI | 26.0% | 95 | agent: mini-SWE-agent | Official leaderboard | Aug 7, 2025 |
| 56 | GPT-4.1 mini | OpenAI | 23.9% | 96 | agent: mini-SWE-agent“gpt-4.1-mini-20250414” | Official leaderboard | Jul 20, 2025 |
| 57 | MCTS-Refine-7B | — | 23.2% | 97 | agent: MCTS-Refine-7B | Official leaderboard | Jun 27, 2025 |
| 58 | GPT-4 Turbo (Nov 2023) | OpenAI | 22.4% | 99 | agent: SWE-agent“gpt-4-1106-preview” | Official leaderboard | Apr 2, 2024 |
| 59 | llama-4-maverick-instruct | Meta | 21.0% | 101 | agent: mini-SWE-agent | Official leaderboard | Jul 20, 2025 |
| 60 | Claude 3 Opus | Anthropic | 15.8% | 102 | agent: SWE-agent“claude-3-opus-20240229” | Official leaderboard | Apr 2, 2024 |
| 61 | gemini-2.0-flash | 13.5% | 103 | agent: mini-SWE-agent | Official leaderboard | Jul 26, 2025 | |
| 62 | llama-4-scout-instruct | Meta | 9.1% | 104 | agent: mini-SWE-agent | Official leaderboard | Jul 20, 2025 |
| 63 | Claude 2 | Anthropic | 4.4% | 107 | agent: RAG baseline“claude-2” | Official leaderboard | Oct 10, 2023 |
| 64 | SWE-Llama | Meta | 1.4% | 109 | agent: RAG baseline | Official leaderboard | Oct 10, 2023 |
Best score over time
Each step is a new best result in our data, by evaluation date.
- 2023-10-104.4%Claude 2
- 2024-04-0222.4%GPT-4 Turbo (Nov 2023)
- 2024-06-1232.6%GPT-4o (May 2024)
- 2024-06-2033.6%Claude 3.5 Sonnet
- 2024-06-2838.4%GPT-4o (May 2024)
- 2024-10-2240.6%Claude 3 Haiku
- 2024-11-0854.2%Claude 3.5 Sonnet
- 2024-12-2162.2%Claude 3.5 Sonnet
- 2025-01-1764.6%o1-preview
- 2025-05-1466.4%Claude 3.7 Sonnet
- 2025-05-2273.2%claude-4-opus
- 2025-07-2074.6%claude-4-sonnet
- 2025-07-3174.8%Claude Sonnet 4
- 2025-08-0476.8%Claude Sonnet 4
- 2025-11-2077.4%Gemini 3 Pro
- 2025-12-0579.2%Claude Opus 4.5
- 2026-04-2083.5%Claude Opus 4.7
Data sources & licences
| Source | Licence / terms | How we reuse it | Last fetched |
|---|---|---|---|
| SWE-bench leaderboards (swebench.com) SWE-bench team, swebench.com leaderboards | No explicit licence (NOASSERTION) – factual scores only, with citation | No explicit licence – factual scores only, with citation | Oct 22, 2026 |
| Epoch AI Benchmarking Hub Epoch AI, 'Capabilities & benchmarking', https://epoch.ai/benchmarks (CC BY 4.0) | CC BY 4.0 | Open licence – full data with attribution | Oct 22, 2026 |
FAQ
What does SWE-bench Verified measure?
500 human-validated GitHub issues from 12 Python repositories; the system must produce a patch that makes the hidden tests pass. % of issues resolved by a model + agent scaffold. Rows from swebench.com are agent systems; rows from Epoch AI use Epoch's own scaffold.
How reliable is SWE-bench Verified?
In February 2026 OpenAI stopped reporting SWE-bench Verified, saying it is contaminated and that 59.4% of 138 audited hard tasks had flawed tests or descriptions. Scores depend heavily on the agent scaffold. Contamination risk: High – OpenAI found frontier models could reproduce gold patches from memory.