Claude 3.5 Haiku features offers enhanced capabilities in speed, coding accuracy, and tool use. Engineered to excel in real-time applications, it delivers quick response times that are essential for dynamic...
Tested on 17 benchmarks with 37.2% average. Top scores: HELM — IFEval (79.2%), HELM — WildBench (76.0%), Lech Mazur Writing (73.5%).
Gemma 3 27B scores 25.1 (101% as good) at $0.08/1M input · 90% cheaper
Computer-aided design evaluation. Tests understanding of CAD concepts, 3D modeling, and engineering design principles.
Unusual and adversarial machine learning challenges. Tests robustness of reasoning about edge cases in ML systems.
Multi-language code editing from Aider. Tests editing ability across Python, JavaScript, TypeScript, Java, C++, Go, Rust, and more.
Stanford HELM WildBench evaluation. Tests reasoning on challenging real-world tasks.
Competition-level math from AMC, AIME, and olympiad problems. Level 5 is the hardest tier, requiring creative problem-solving.
Stanford HELM evaluation of mathematical reasoning across diverse problem types.
Mock AIME (American Invitational Mathematics Exam) problems from OTIS. Tests mathematical competition performance.
- Typemultimodal
- Context200K tokens (~100 books)
- ReleasedNov 2024
- LicenseProprietary
- StatusActive
- Cost / Message~$0.006