Compare · ModelsLive · 2 picked · head to head
Claude 3.5 Sonnet vs o1
Side by side · benchmarks, pricing, and signals you can act on.
Winner summary
o1 wins on 13/15 benchmarks
o1 wins 13 of 15 shared benchmarks. Leads in coding · general · math.
Category leads
coding·o1general·o1math·o1knowledge·o1reasoning·o1
Hype vs Reality
Attention vs performance
Claude 3.5 Sonnet
#203 by perf·no signal
o1
#171 by perf·no signal
Vendor risk
Who is behind the model
Anthropic
$965.0B·Tier 1
OpenAI
$840.0B·Tier 1
Head to head
15 benchmarks · 2 models
Claude 3.5 Sonneto1
Aider · Code Editing
Claude 3.5 Sonnet
84.2
o1
84.2
Aider polyglot
o1 leads by +10.1
Aider Polyglot · measures how well AI models can edit code across multiple programming languages using the Aider coding assistant framework.
Claude 3.5 Sonnet
51.6
o1
61.7
CadEval
o1 leads by +8.0
CadEval · evaluates the ability to generate and reason about Computer-Aided Design code, testing spatial reasoning and engineering knowledge.
Claude 3.5 Sonnet
48.0
o1
56.0
Dtbench
o1 leads by +11.5
Claude 3.5 Sonnet
46.3
o1
57.8
FrontierMath-2025-02-28-Private
o1 leads by +7.5
FrontierMath (Feb 2025) · original research-level math problems created by mathematicians, testing capabilities at the boundary of current AI mathematical reasoning.
Claude 3.5 Sonnet
1.8
o1
9.3
GeoBench
o1 leads by +18.0
GeoBench · tests geographic knowledge and spatial reasoning across countries, landmarks, coordinates, and geopolitical understanding.
Claude 3.5 Sonnet
62.0
o1
80.0
GPQA diamond
o1 leads by +33.2
Graduate-Level Google-Proof QA (Diamond set) · expert-crafted questions in physics, biology, and chemistry that are difficult even for domain PhDs.
Claude 3.5 Sonnet
38.7
o1
71.9
HLE
o1 leads by +3.3
HLE (Humanity's Last Exam) · a reasoning benchmark designed to be the hardest public evaluation of AI. Questions span mathematics, physics, philosophy, and logic · curated to be at or beyond the frontier of human expert capability. Tested with and without tool augmentation.
Claude 3.5 Sonnet
0.0
o1
3.3
Lech Mazur Writing
Claude 3.5 Sonnet leads by +10.1
Lech Mazur Writing · evaluates creative writing ability, assessing prose quality, narrative coherence, and stylistic sophistication.
Claude 3.5 Sonnet
80.3
o1
70.2
MATH level 5
o1 leads by +43.0
MATH Level 5 · the hardest tier of the MATH benchmark, featuring competition-level problems from AMC, AIME, and Olympiad-style mathematics.
Claude 3.5 Sonnet
51.7
o1
94.7
Metr Time Horizons
o1 leads by +10.9
Claude 3.5 Sonnet
40.1
o1
51.0
OTIS Mock AIME 2024-2025
o1 leads by +66.9
OTIS Mock AIME 2024-2025 · simulated American Invitational Mathematics Examination problems testing advanced problem-solving skills.
Claude 3.5 Sonnet
6.4
o1
73.3
SimpleBench
o1 leads by +15.1
SimpleBench · tests fundamental reasoning capabilities with straightforward problems designed to expose gaps in basic logical and spatial thinking.
Claude 3.5 Sonnet
13.0
o1
28.1
VPCT
o1 leads by +5.5
VPCT (Visual Pattern Completion Test) · tests visual reasoning and pattern recognition by having models complete visual sequences and transformations.
Claude 3.5 Sonnet
0.0
o1
5.5
WeirdML
o1 leads by +12.9
WeirdML · tests models on unusual and adversarial machine learning tasks that require creative problem-solving beyond standard patterns.
Claude 3.5 Sonnet
31.0
o1
43.8
Full benchmark table
| Benchmark | Claude 3.5 Sonnet | o1 |
|---|---|---|
Aider · Code Editing | 84.2 | 84.2 |
Aider polyglot Aider Polyglot · measures how well AI models can edit code across multiple programming languages using the Aider coding assistant framework. | 51.6 | 61.7 |
CadEval CadEval · evaluates the ability to generate and reason about Computer-Aided Design code, testing spatial reasoning and engineering knowledge. | 48.0 | 56.0 |
Dtbench | 46.3 | 57.8 |
FrontierMath-2025-02-28-Private FrontierMath (Feb 2025) · original research-level math problems created by mathematicians, testing capabilities at the boundary of current AI mathematical reasoning. | 1.8 | 9.3 |
GeoBench GeoBench · tests geographic knowledge and spatial reasoning across countries, landmarks, coordinates, and geopolitical understanding. | 62.0 | 80.0 |
GPQA diamond Graduate-Level Google-Proof QA (Diamond set) · expert-crafted questions in physics, biology, and chemistry that are difficult even for domain PhDs. | 38.7 | 71.9 |
HLE HLE (Humanity's Last Exam) · a reasoning benchmark designed to be the hardest public evaluation of AI. Questions span mathematics, physics, philosophy, and logic · curated to be at or beyond the frontier of human expert capability. Tested with and without tool augmentation. | 0.0 | 3.3 |
Lech Mazur Writing Lech Mazur Writing · evaluates creative writing ability, assessing prose quality, narrative coherence, and stylistic sophistication. | 80.3 | 70.2 |
MATH level 5 MATH Level 5 · the hardest tier of the MATH benchmark, featuring competition-level problems from AMC, AIME, and Olympiad-style mathematics. | 51.7 | 94.7 |
Metr Time Horizons | 40.1 | 51.0 |
OTIS Mock AIME 2024-2025 OTIS Mock AIME 2024-2025 · simulated American Invitational Mathematics Examination problems testing advanced problem-solving skills. | 6.4 | 73.3 |
SimpleBench SimpleBench · tests fundamental reasoning capabilities with straightforward problems designed to expose gaps in basic logical and spatial thinking. | 13.0 | 28.1 |
VPCT VPCT (Visual Pattern Completion Test) · tests visual reasoning and pattern recognition by having models complete visual sequences and transformations. | 0.0 | 5.5 |
WeirdML WeirdML · tests models on unusual and adversarial machine learning tasks that require creative problem-solving beyond standard patterns. | 31.0 | 43.8 |
Pricing · per 1M tokens · projected $/mo at 10M tokens
| Model | Input | Output | Context | Projected $/mo |
|---|---|---|---|---|
| n/a | n/a | n/a | n/a | |
| $15.00 | $60.00 | 200K tokens (~100 books) | $262.50 |