Benchmarks/WeirdML

WeirdML

WeirdML — tests models on unusual and adversarial machine learning tasks that require creative problem-solving beyond standard patterns.

coding

Models Tested

77.9

Top Score

37.8

Average Score

Rankings

#	Model	Score
1	Claude Opus 4.6Anthropic	77.9
2	GPT-5.2 ChatOpenAI	72.2
3	GPT-5.2OpenAI	72.2
4	Gemini 3.1 Pro PreviewGoogle DeepMind	72.1
5	Gemini 3 ProGoogle	69.9
6	Claude Sonnet 4.6Anthropic	66.1
7	Claude Opus 4.5Anthropic	63.7
8	Gemini 3 Flash PreviewGoogle DeepMind	61.6
9	GPT-5.1OpenAI	60.8
10	GPT-5.1 ChatOpenAI	60.8
11	GPT-5 ChatOpenAI	60.7
12	GPT-5OpenAI	60.7
13	GPT-5 ProOpenAI	60.4
14	o3 ProOpenAI	58.2
15	GPT-5.4OpenAI	57.4
16	Gemini 2.5 ProGoogle DeepMind	54.0
17	GPT-5 MiniOpenAI	52.7
18	o4 MiniOpenAI	52.6
19	o3OpenAI	52.4
20	ZA GLM 5 Turboz-ai	48.2
21	ZA GLM 5z-ai	48.2
22	gpt-oss-120b (free)OpenAI	48.2
23	gpt-oss-120bOpenAI	48.2
24	gpt-oss-20b (free)OpenAI	48.2
25	gpt-oss-20bOpenAI	48.2
26	Claude Sonnet 4.5Anthropic	47.7
27	o1-previewOpenAI	47.6
28	Claude Sonnet 4Anthropic	46.1
29	Grok 4xAI	45.7
30	M Kimi K2.5moonshotai	45.6
31	Claude Haiku 4.5Anthropic	45.4
32	o1OpenAI	43.8
33	o3 MiniOpenAI	43.7
34	Claude Opus 4Anthropic	43.4
35	Grok 4 FastxAI	42.9
36	M Kimi K2 Thinkingmoonshotai	42.8
37	Claude Opus 4.1Anthropic	42.8
38	Grok 3 MinixAI	42.6
39	Grok 3 Mini BetaxAI	42.6
40	DeepSeek-R1 (May 2025)DeepSeek	41.6
41	Qwen3 Coder 480B A35B (free)Alibaba Qwen	41.2
42	Qwen3 Coder 480B A35BAlibaba Qwen	41.2
43	Qwen3 235B A22B Thinking 2507Alibaba Qwen	41.0
44	Gemini 2.5 FlashGoogle DeepMind	41.0
45	ZA GLM 4.5z-ai	40.6
46	DeepSeek V3.2 ExpDeepSeek	39.5
47	GPT-4.5OpenAI	39.4
48	M Kimi K2 0711moonshotai	39.4
49	GPT-4.1OpenAI	39.0
50	Qwen3-235B-A22B-Instruct (Jul 2025)Alibaba	38.7
51	DeepSeek V3.1DeepSeek	38.4
52	GPT-5 NanoOpenAI	38.1
53	GPT-4.1 MiniOpenAI	37.6
54	Qwen3 235B A22BAlibaba Qwen	37.3
55	Grok 3xAI	37.2
56	Grok 3 BetaxAI	37.2
57	M Kimi K2 0905moonshotai	36.7
58	DeepSeek-R1DeepSeek	36.5
59	o1-miniOpenAI	36.3
60	Claude 3.5 SonnetAnthropic	31.0
61	Claude 3.5 HaikuAnthropic	30.7
62	GPT-4o (2024-11-20)OpenAI	25.1
63	Gemini 1.5 Flash (Sep 2024)Google	24.9
64	Llama 4 MaverickMeta	24.5
65	Claude 3 OpusAnthropic	23.2
66	Grok-2 (Dec 2024)xAI	22.2
67	Gemini 1.5 Pro (Sept 2024)Google	22.2
68	Llama 3.1-405BMeta	21.4
69	GPT-4.1 NanoOpenAI	19.0
70	GPT-4 Turbo PreviewOpenAI	18.0
71	Llama 3.3 70B Instruct (free)Meta	14.4
72	Llama 3.3 70B InstructMeta	14.4
73	GPT-4 TurboOpenAI	12.4
74	GPT-4 Turbo (older v1106)OpenAI	12.4
75	GPT-4 (older v0314)OpenAI	12.4
76	GPT-4OpenAI	12.4
77	GPT-4o-mini (2024-07-18)OpenAI	11.8
78	GPT-4o-miniOpenAI	11.8
79	Claude 3 SonnetAnthropic	10.2
80	Claude 3 HaikuAnthropic	9.8
81	Llama 3.1 70B InstructMeta	9.0
82	Claude 2.1Anthropic	7.1
83	GPT-3.5 Turbo (older v0613)OpenAI	3.5
84	GPT-3.5 Turbo InstructOpenAI	3.5
85	GPT-3.5 TurboOpenAI	3.5
86	Mixtral 8x22B InstructMistral AI	3.2
87	Llama 3.1 8B InstructMeta	1.7

Related Benchmarks

SWE-Bench Verified (Bash Only)

32 models

Terminal Bench

27 models