MMLU Pro

MMLU Pro was released in 2024 to address the saturation of the original MMLU benchmark. Frontier models had pushed MMLU scores above 90%, compressing the range where models could differentiate. MMLU Pro uses 12,000+ questions across 14 subjects, with 10 answer choices per question instead of 4 and a heavier emphasis on multi-step reasoning. Top of the board sits around 78-82% in 2026.

MMLU Pro filters trivial and ambiguous questions from the original MMLU and adds new reasoning-heavy items. Each question has 10 options to reduce lucky-guess inflation from 25% to 10% baseline. Subject coverage: Math, Physics, Chemistry, Biology, Computer Science, Engineering, Economics, Business, Law, Psychology, Philosophy, History, Health, and Other. The "Math" subset is the hardest · top models score below 70%. Pro reports significantly larger gaps between frontier and mid-tier models than plain MMLU, making it the better discriminator in 2026.

Construction methodology: Start with the full MMLU corpus. Filter using multiple expert annotators for quality (remove ambiguous, trivial, or label-noise items). Regenerate answer choices with distractor generation methods that use semantic similarity + factual plausibility. Validate the 10-choice format preserves separability. Hard subset: questions where majority of frontier models answered incorrectly or diverged · concentrates discriminating power. Frontier correlation: MMLU Pro scores correlate 0.85+ with Chatbot Arena rankings, better than MMLU.