Agentic coding · %
SWE-bench Pro
Scale AI's harder, longer-horizon successor to SWE-bench with a 731-task public split and private/commercial splits.
What it measures
% of tasks resolved.
Limitations
Not collected (no open data file). OpenAI first recommended it in place of Verified, then retracted the recommendation after estimating that about 30% of Pro tasks are broken.
Contamination risk: Medium
Private splits limit contamination; the public split is exposed.
We do not copy this leaderboard (no open data, or the terms do not allow it). Please see the official page for current results.
https://scale.com/leaderboard/swe_bench_pro_public ↗Data sources & licences
| Source | Licence / terms | How we reuse it | Last fetched |
|---|---|---|---|
| Scale AI SEAL leaderboards (SWE-bench Pro, HLE) Scale AI | No open data interface | Not collected – link only | — |
FAQ
What does SWE-bench Pro measure?
Scale AI's harder, longer-horizon successor to SWE-bench with a 731-task public split and private/commercial splits. % of tasks resolved.
How reliable is SWE-bench Pro?
Not collected (no open data file). OpenAI first recommended it in place of Verified, then retracted the recommendation after estimating that about 30% of Pro tasks are broken. Contamination risk: Medium – Private splits limit contamination; the public split is exposed.