CMMLU
The Frontier
Best score over time · one chart, every benchmark
Full rankings
9 models tested · sorted by score
| # | Model | Score |
|---|---|---|
| 1 | 89.7 | |
| 2 | 85.7 | |
| 3 | 71.0 | |
| 4 | 71.0 | |
| 5 | 64.4 | |
| 6 | 58.7 | |
| 7 | 41.5 | |
| 8 | 39.8 | |
| 9 | 36.9 |
Score distribution
Where models cluster
Correlated benchmarks
Pearson r · original research
Benchmarks that track with CMMLU
Pearson correlation across models scored on both benchmarks. Closer to 1 = strongly predictive.
Frequently asked
About CMMLU
What does CMMLU measure?
CMMLU is a knowledge benchmark in the BenchGecko catalog. 9 AI models have been tested on it. Scores range from 36.9 to 89.7 out of 100.
Which model leads on CMMLU?
Qwen2-72B from Alibaba Qwen leads CMMLU with a score of 89.7. The median score across 9 tested models is 64.4.
Is CMMLU saturated?
No · the top score is 89.7 out of 100 (90%). There is still meaningful room for improvement on CMMLU.
Does CMMLU predict performance on other benchmarks?
Yes · CMMLU scores correlate 0.98 with BBH across 6 shared models. Models that do well on CMMLU tend to do well on BBH.
How often is CMMLU data refreshed?
BenchGecko pulls updates daily. New model scores on CMMLU appear as soon as they are published by Epoch AI or the model provider.
- Category
- Knowledge
- Max score
- 100
- Models
- 9
- Updated
- 2024-09-19
Top on CMMLU
Qwen2-72B · 89.7Qwen2.5 72B Instruct · 85.7GPT-4 Turbo (batch) · 71.0GPT-4 Turbo · 71.0Llama 3.1 70B Instruct · 64.4More knowledge benchmarks
Same category · related evaluations