PIQA

PIQA (Physical Interaction QA) — tests intuitive physical reasoning by asking models to select the correct approach for everyday physical tasks.

knowledge

Models Tested

77.4

Top Score

63.2

Average Score

Rankings

#	Model	Score
1	GPT-4o-mini (2024-07-18)OpenAI	77.4
2	GPT-4o-miniOpenAI	77.4
3	Gemini 1.5 Flash (Sep 2024)Google	75.0
4	Llama 3.1-405BMeta	71.8
5	T Falcon-180BTII	69.8
6	DeepSeek V3DeepSeek	69.4
7	DeepSeek-V2 (MoE-236B, May 2024)DeepSeek	67.8
8	Gemma 2 9BGoogle DeepMind	67.4
9	Mixtral 8x7B InstructMistral AI	67.2
10	Mistral NemoMistral AI	67.0
11	U Stable Beluga 2unknown	66.6
12	Mistral 7B v0.1Mistral	66.0
13	T Falcon-40BTII	66.0
14	LLaMA-65BMeta	65.6
15	Llama 2-70BMeta	65.6
16	Qwen2.5 72B InstructAlibaba Qwen	65.2
17	U Nemotron-4 15Bunknown	64.8
18	LLaMA-33BMeta	64.6
19	Llama 2-34BMeta	63.8
20	U MPT-30Bunknown	63.8
21	Llama 3.1 8B InstructMeta	62.4
22	Gemma 7BGoogle	62.4
23	Llama 2-13BMeta	61.6
24	U MPT-7Bunknown	61.2
25	T Falcon-7BTII	60.6
26	LLaMA-13BMeta	60.2
27	Qwen-14BAlibaba	59.8
28	LLaMA-7BMeta	59.6
29	Llama 2-7BMeta	57.6
30	U Baichuan2-13Bunknown	56.2
31	Qwen-7BAlibaba	55.8
32	Gemma 2BGoogle	54.6
33	U Baichuan1-7Bunknown	52.4
34	U XGen-7Bunknown	51.0
35	U Dolly 2.0-12bunknown	50.8
36	Cerebras-GPT-13BOpenAI	47.0