Compare · ModelsLive · 2 picked · head to head
Gemma 2B vs GPT-4o-mini (2024-07-18)
Side by side · benchmarks, pricing, and signals you can act on.
Winner summary
GPT-4o-mini (2024-07-18) wins on 3/3 benchmarks
GPT-4o-mini (2024-07-18) wins 3 of 3 shared benchmarks. Leads in math · knowledge.
Category leads
math·GPT-4o-mini (2024-07-18)knowledge·GPT-4o-mini (2024-07-18)
Hype vs Reality
Attention vs performance
Gemma 2B
#255 by perf·#7 by attention
GPT-4o-mini (2024-07-18)
#171 by perf·no signal
Best value
GPT-4o-mini (2024-07-18)
Gemma 2B
n/a
no price
GPT-4o-mini (2024-07-18)
115.2 pts/$
$0.38/M
Vendor risk
Who is behind the model
Google DeepMind
$4.20T·Tier 1
OpenAI
$840.0B·Tier 1
Head to head
3 benchmarks · 2 models
Gemma 2BGPT-4o-mini (2024-07-18)
GSM8K
GPT-4o-mini (2024-07-18) leads by +73.6
Grade School Math 8K · 8,500 linguistically diverse grade-school math word problems that require multi-step reasoning to solve.
Gemma 2B
17.7
GPT-4o-mini (2024-07-18)
91.3
MMLU
GPT-4o-mini (2024-07-18) leads by +52.7
Massive Multitask Language Understanding · 57 subjects spanning STEM, humanities, social sciences, and more. The standard benchmark for broad knowledge.
Gemma 2B
23.1
GPT-4o-mini (2024-07-18)
75.7
PIQA
GPT-4o-mini (2024-07-18) leads by +22.8
PIQA (Physical Interaction QA) · tests intuitive physical reasoning by asking models to select the correct approach for everyday physical tasks.
Gemma 2B
54.6
GPT-4o-mini (2024-07-18)
77.4
Full benchmark table
| Benchmark | Gemma 2B | GPT-4o-mini (2024-07-18) |
|---|---|---|
GSM8K Grade School Math 8K · 8,500 linguistically diverse grade-school math word problems that require multi-step reasoning to solve. | 17.7 | 91.3 |
MMLU Massive Multitask Language Understanding · 57 subjects spanning STEM, humanities, social sciences, and more. The standard benchmark for broad knowledge. | 23.1 | 75.7 |
PIQA PIQA (Physical Interaction QA) · tests intuitive physical reasoning by asking models to select the correct approach for everyday physical tasks. | 54.6 | 77.4 |
Pricing · per 1M tokens · projected $/mo at 10M tokens
| Model | Input | Output | Context | Projected $/mo |
|---|---|---|---|---|
| — | — | — | — | |
| $0.15 | $0.60 | 128K tokens (~64 books) | $2.62 |