Benchmarks/SWE-Bench Verified (Bash Only)

SWE-Bench Verified (Bash Only)

SWE-Bench Verified (Bash Only) — a curated subset of SWE-bench where models fix real Python repository bugs using only bash commands, no agent frameworks.

coding

Models Tested

74.4

Top Score

49.4

Average Score

Rankings

#	Model	Score
1	Claude Opus 4.5Anthropic	74.4
2	Gemini 3 ProGoogle	74.2
3	GPT-5.2 ChatOpenAI	71.8
4	GPT-5.2OpenAI	71.8
5	Claude Sonnet 4.5Anthropic	70.6
6	Claude Opus 4Anthropic	67.6
7	GPT-5.1OpenAI	66.0
8	GPT-5.1 ChatOpenAI	66.0
9	GPT-5 ChatOpenAI	65.0
10	GPT-5OpenAI	65.0
11	Claude Sonnet 4Anthropic	64.9
12	M Kimi K2 Thinkingmoonshotai	63.4
13	GPT-5 MiniOpenAI	59.8
14	o3OpenAI	58.4
15	Qwen3 Coder 480B A35B (free)Alibaba Qwen	55.4
16	Qwen3 Coder 480B A35BAlibaba Qwen	55.4
17	ZA GLM 4.5z-ai	54.2
18	Claude 3.7 SonnetAnthropic	52.8
19	Claude 3.7 Sonnet (thinking)Anthropic	52.8
20	o4 MiniOpenAI	45.0
21	M Kimi K2 0711moonshotai	43.8
22	GPT-4.1OpenAI	39.6
23	GPT-5 NanoOpenAI	34.8
24	Gemini 2.5 FlashGoogle DeepMind	28.7
25	gpt-oss-120b (free)OpenAI	26.0
26	gpt-oss-120bOpenAI	26.0
27	gpt-oss-20b (free)OpenAI	26.0
28	gpt-oss-20bOpenAI	26.0
29	GPT-4.1 MiniOpenAI	23.9
30	GPT-4o (2024-11-20)OpenAI	21.6
31	Llama 4 MaverickMeta	21.0
32	Llama 4 ScoutMeta	9.1

Related Benchmarks