Benchmarks/MMLU

MMLU

Massive Multitask Language Understanding — 57 subjects spanning STEM, humanities, social sciences, and more. The standard benchmark for broad knowledge.

knowledge

Models Tested

84.1

Top Score

55.4

Average Score

Rankings

#	Model	Score
1	GPT-4o (2024-11-20)OpenAI	84.1
2	DeepSeek V3DeepSeek	82.9
3	Gemini 1.5 Pro (Sept 2024)Google	82.5
4	Claude 3.5 SonnetAnthropic	82.0
5	Llama 3.3 70B Instruct (free)Meta	81.7
6	Llama 3.3 70B InstructMeta	81.7
7	Qwen2.5 72B InstructAlibaba Qwen	80.4
8	Phi 4Microsoft	79.7
9	Claude 3 OpusAnthropic	79.5
10	Llama 3.1-405BMeta	79.3
11	GPT-4o (2024-08-06)OpenAI	79.1
12	GPT-4o (2024-05-13)OpenAI	79.1
13	GPT-4oOpenAI	79.1
14	GPT-4o (extended)OpenAI	79.1
15	Gemini 1.5 Pro (Feb 2024)Google	76.9
16	GPT-4 TurboOpenAI	76.5
17	GPT-4 Turbo (older v1106)OpenAI	76.5
18	GPT-4 (older v0314)OpenAI	76.5
19	GPT-4OpenAI	76.5
20	Qwen2-72BAlibaba	76.5
21	GPT-4o-mini (2024-07-18)OpenAI	75.7
22	GPT-4o-miniOpenAI	75.7
23	GPT-4 Turbo PreviewOpenAI	75.1
24	Llama 3.2 90BMeta	73.7
25	Llama 3.1 70B InstructMeta	73.5
26	Mistral Large 2407Mistral AI	73.3
27	Gemini 2.0 FlashGoogle DeepMind	72.9
28	Llama 3 70B InstructMeta	72.4
29	Qwen2.5 Coder 32B InstructAlibaba Qwen	72.1
30	Claude 2Anthropic	71.3
31	DeepSeek-V2 (MoE-236B, May 2024)DeepSeek	71.2
32	phi-3-medium 14BMicrosoft	70.7
33	Gemini 1.5 Flash (May 2024)Google	70.5
34	Mixtral 8x22B InstructMistral AI	70.4
35	U Yi-34Bunknown	68.4
36	Claude 3 SonnetAnthropic	67.9
37	Gemma 2 27BGoogle DeepMind	67.6
38	phi-3-small 7.4BMicrosoft	67.6
39	Claude 3.5 HaikuAnthropic	65.7
40	Gemini 1.5 Flash (Sep 2024)Google	65.2
41	Claude 3 HaikuAnthropic	65.1
42	Claude 2.1Anthropic	64.7
43	Claude InstantAnthropic	64.5
44	Gemma 2 9BGoogle DeepMind	62.8
45	Mixtral 8x7B InstructMistral AI	60.8
46	T Falcon-180BTII	60.8
47	Gemini 1.0 ProGoogle	60.0
48	Llama 2-70BMeta	59.9
49	Llama 3 8B InstructMeta	58.4
50	Mistral LargeMistral AI	58.4
51	phi-3-mini 3.8BMicrosoft	58.4
52	U Stable Beluga 2unknown	58.1
53	Qwen2.5 Coder 7B InstructAlibaba Qwen	57.3
54	GPT-3.5 Turbo (older v0613)OpenAI	56.4
55	GPT-3.5 Turbo InstructOpenAI	56.4
56	GPT-3.5 TurboOpenAI	56.4
57	Qwen-14BAlibaba	55.1
58	Gemma 7BGoogle	54.8
59	U StarCoder 2 15Bunknown	52.1
60	U Yi 6Bunknown	52.0
61	LLaMA-65BMeta	51.2
62	Llama 2-34BMeta	50.1
63	Mistral 7B v0.1Mistral	50.0
64	U Baichuan2-13Bunknown	45.6
65	LLaMA-33BMeta	44.9
66	U Nemotron-4 15Bunknown	44.9
67	Phi-2Microsoft	44.5
68	T Falcon 2 11BTII	44.5
69	T Falcon-40BTII	42.5
70	Llama 3.1 8B InstructMeta	41.5
71	Llama 2-13BMeta	40.8
72	U Baichuan 2-7Bunknown	38.9
73	Qwen2.5-Coder (1.5B)Alibaba	38.1
74	U INTELLECT-1unknown	33.2
75	U MPT-30Bunknown	30.5
76	LLaMA-13BMeta	30.3
77	Llama 2-7BMeta	27.7
78	Qwen-7BAlibaba	26.7
79	U Baichuan1-7Bunknown	23.1
80	Gemma 2BGoogle	23.1
81	DeepSeek Coder 33BDeepSeek	19.2
82	U StarCoder 2 7Bunknown	18.4
83	Phi-1.5Microsoft	16.8
84	U StarCoder 2 3Bunknown	15.5
85	DeepSeek Coder 6.7BDeepSeek	15.2
86	U XGen-7Bunknown	15.1
87	LLaMA-7BMeta	14.1
88	T Falcon-7BTII	13.3
89	U MPT-7Bunknown	7.7
90	U Dolly 2.0-12bunknown	1.6
91	Cerebras-GPT-13BOpenAI	1.6
92	DeepSeek Coder 1.3BDeepSeek	1.1

Related Benchmarks