Chinese · %
CMMLU
Chinese multitask knowledge benchmark: 67 subjects including China-specific topics.
Not updated recently Contamination risk: High 20 results · 12 models Updated Oct 22, 2026 Newest result: 2024-12-06 · stale Official leaderboard ↗
What it measures
Average accuracy (five-shot / zero-shot).
Limitations
Leaderboard last updated in 2024; mostly older models; data licence is non-commercial, so only the top entries are cited.
Contamination risk: High
Public test set.
Leaderboard · Subset / setting: five-shot
10 models
| # | Model | Provider | Score | Rank | Setting | Source type | Date |
|---|---|---|---|---|---|---|---|
| 1 | Lingzhi-72B-chat | — | 90.3% | 1 | Five-shot average | Official leaderboard | Dec 6, 2024 |
| 2 | Telechat2-35B | — | 90.2% | 2 | Five-shot average | Official leaderboard | Dec 6, 2024 |
| 3 | Spark 4.0-2024-10-14 | iFlytek | 90.1% | 3 | Five-shot average | Official leaderboard | Dec 6, 2024 |
| 4 | Qwen2-72BOW | Alibaba (Qwen) | 89.7% | 4 | Five-shot average | Official leaderboard | Dec 6, 2024 |
| 5 | Jiutian-大模型 | — | 88.6% | 5 | Five-shot average | Official leaderboard | Dec 6, 2024 |
| 6 | Qwen1.5-110BOW | Alibaba (Qwen) | 88.3% | 6 | Five-shot average | Official leaderboard | Dec 6, 2024 |
| 7 | JIUTIAN-57B | — | 87.4% | 7 | Five-shot average | Official leaderboard | Dec 6, 2024 |
| 8 | Qwen2.5-72BOW | Alibaba (Qwen) | 85.7% | 8 | Five-shot average | Official leaderboard | Dec 6, 2024 |
| 9 | Youyuanjian | — | 83.7% | 9 | Five-shot average | Official leaderboard | Dec 6, 2024 |
| 10 | Qwen1.5-72BOW | Alibaba (Qwen) | 83.5% | 10 | Five-shot average | Official leaderboard | Dec 6, 2024 |
Leaderboard · Subset / setting: zero-shot
10 models
| # | Model | Provider | Score | Rank | Setting | Source type | Date |
|---|---|---|---|---|---|---|---|
| 1 | Spark 4.0-2024-10-14 | iFlytek | 91.0% | 1 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
| 2 | Telechat2-35B | — | 90.5% | 2 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
| 3 | Lingzhi-72B-chat | — | 90.1% | 3 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
| 4 | Qwen1.5-110BOW | Alibaba (Qwen) | 87.6% | 4 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
| 5 | Qwen2-72BOW | Alibaba (Qwen) | 87.5% | 5 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
| 6 | Qwen2.5-72BOW | Alibaba (Qwen) | 84.7% | 6 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
| 7 | PCI-TransGPT | — | 82.4% | 7 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
| 8 | Qwen1.5-72BOW | Alibaba (Qwen) | 81.8% | 8 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
| 9 | Qwen1.5-32BOW | Alibaba (Qwen) | 81.5% | 9 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
| 10 | Youyuanjian | — | 81.2% | 10 | Zero-shot average | Official leaderboard | Dec 6, 2024 |
Data sources & licences
| Source | Licence / terms | How we reuse it | Last fetched |
|---|---|---|---|
| CMMLU leaderboard (GitHub README) Li et al., CMMLU (CC BY-NC-SA 4.0) | CC BY-NC-SA 4.0 | Restricted – only top entries cited | Oct 22, 2026 |
FAQ
What does CMMLU measure?
Chinese multitask knowledge benchmark: 67 subjects including China-specific topics. Average accuracy (five-shot / zero-shot).
How reliable is CMMLU?
Leaderboard last updated in 2024; mostly older models; data licence is non-commercial, so only the top entries are cited. Contamination risk: High – Public test set.