ARC AI2

AI2 Reasoning Challenge — tests grade-school level science knowledge with multiple-choice questions requiring reasoning beyond simple retrieval.

knowledge

Models Tested

93.7

Top Score

48.8

Average Score

Rankings

#	Model	Score
1	DeepSeek V3DeepSeek	93.7
2	Llama 3.1-405BMeta	93.7
3	Qwen2.5 72B InstructAlibaba Qwen	92.7
4	DeepSeek-V2 (MoE-236B, May 2024)DeepSeek	89.6
5	phi-3-medium 14BMicrosoft	88.8
6	phi-3-small 7.4BMicrosoft	87.6
7	Mixtral 8x7B InstructMistral AI	83.1
8	Claude InstantAnthropic	81.7
9	U Stable Beluga 2unknown	81.5
10	phi-3-mini 3.8BMicrosoft	79.9
11	Qwen-14BAlibaba	79.2
12	Llama 3 8B InstructMeta	77.1
13	Mistral 7B v0.1Mistral	71.5
14	Llama 2-70BMeta	71.1
15	Gemma 7BGoogle	71.1
16	Phi-2Microsoft	67.9
17	Qwen-7BAlibaba	67.1
18	Qwen2.5 Coder 32B InstructAlibaba Qwen	60.7
19	LLaMA-65BMeta	59.3
20	T Falcon-180BTII	57.1
21	LLaMA-33BMeta	56.7
22	T Falcon-40BTII	49.1
23	Qwen2.5 Coder 7B InstructAlibaba Qwen	47.9
24	Llama 2-13BMeta	47.1
25	U Nemotron-4 15Bunknown	40.7
26	U INTELLECT-1unknown	39.4
27	Llama 2-34BMeta	39.3
28	LLaMA-13BMeta	36.9
29	U MPT-30Bunknown	34.1
30	U Yi 6Bunknown	33.7
31	T Falcon-7BTII	30.5
32	LLaMA-7BMeta	30.1
33	U StarCoder 2 15Bunknown	29.6
34	Llama 2-7BMeta	27.9
35	Qwen2.5-Coder (1.5B)Alibaba	26.9
36	Phi-1.5Microsoft	25.9
37	U MPT-7Bunknown	23.5
38	DeepSeek Coder 33BDeepSeek	22.9
39	Gemma 2BGoogle	22.8
40	U XGen-7Bunknown	21.6
41	U Dolly 2.0-12bunknown	19.5
42	U StarCoder 2 7Bunknown	18.3
43	U Baichuan2-13Bunknown	17.3
44	DeepSeek Coder 6.7BDeepSeek	15.2
45	U StarCoder 2 3Bunknown	12.3
46	U Baichuan 2-7Bunknown	10.0
47	Cerebras-GPT-13BOpenAI	9.9
48	DeepSeek Coder 1.3BDeepSeek	0.5