Benchmarks/Winogrande

Winogrande

WinoGrande — large-scale commonsense reasoning benchmark where models must resolve ambiguous pronouns in carefully constructed sentence pairs.

knowledge

Models Tested

78.4

Top Score

45.2

Average Score

Rankings

#	Model	Score
1	Llama 3.1-405BMeta	78.4
2	Claude 3 OpusAnthropic	77.0
3	T Falcon-180BTII	74.2
4	DeepSeek-V2 (MoE-236B, May 2024)DeepSeek	72.6
5	DeepSeek V3DeepSeek	70.4
6	Llama 3 70B InstructMeta	67.0
7	Qwen2.5 72B InstructAlibaba Qwen	64.6
8	phi-3-medium 14BMicrosoft	63.0
9	phi-3-small 7.4BMicrosoft	63.0
10	Qwen2.5 Coder 32B InstructAlibaba Qwen	61.6
11	Llama 2-70BMeta	60.4
12	Gemma 7BGoogle	58.0
13	T Falcon 2 11BTII	56.6
14	U Nemotron-4 15Bunknown	56.0
15	Mixtral 8x7B InstructMistral AI	54.4
16	LLaMA-65BMeta	54.0
17	T Falcon-40BTII	53.8
18	Llama 2-34BMeta	53.4
19	LLaMA-33BMeta	52.0
20	Llama 3 8B InstructMeta	51.4
21	Mistral 7B v0.1Mistral	50.6
22	Claude 3 SonnetAnthropic	50.2
23	Claude 3 HaikuAnthropic	48.4
24	Phi-1.5Microsoft	46.8
25	LLaMA-13BMeta	46.0
26	Qwen2.5 Coder 7B InstructAlibaba Qwen	45.8
27	Llama 2-13BMeta	45.6
28	U Yi 6Bunknown	42.6
29	U MPT-30Bunknown	42.0
30	phi-3-mini 3.8BMicrosoft	41.6
31	LLaMA-7BMeta	40.2
32	Llama 2-7BMeta	38.4
33	U MPT-7Bunknown	37.2
34	T Falcon-7BTII	34.4
35	U INTELLECT-1unknown	31.6
36	Gemma 2BGoogle	30.8
37	U XGen-7Bunknown	29.8
38	U StarCoder 2 15Bunknown	28.6
39	DeepSeek Coder 33BDeepSeek	24.0
40	U Dolly 2.0-12bunknown	23.6
41	Cerebras-GPT-13BOpenAI	21.6
42	Qwen2.5-Coder (1.5B)Alibaba	21.4
43	DeepSeek Coder 6.7BDeepSeek	15.2
44	U StarCoder 2 3Bunknown	14.2
45	U StarCoder 2 7Bunknown	14.2
46	Phi-2Microsoft	9.4
47	DeepSeek Coder 1.3BDeepSeek	6.6