Tested on 6 benchmarks with 50.4% average. Top scores: Fiction.LiveBench (94.4%), Lech Mazur Writing (81.1%), ARC-AGI (48.5%).
Unusual and adversarial machine learning challenges. Tests robustness of reasoning about edge cases in ML systems.
Capture-the-flag cybersecurity challenges. Tests vulnerability analysis, reverse engineering, cryptography, and exploitation skills.
Abstraction and Reasoning Corpus. Tests fluid intelligence through novel visual pattern recognition puzzles. Core measure of general intelligence.
ARC-AGI 2, harder sequel to ARC. More complex abstract reasoning patterns that test generalization ability beyond training data.
LiveBench fiction analysis. Tests literary comprehension and creative text understanding.
Writing quality evaluation by Lech Mazur. Tests prose quality, coherence, and stylistic ability.
- Typetext
- ContextN/A
- ReleasedJan 2024
- LicenseProprietary
- Statusbenchmark-only