Chatbot Arena

Chatbot Arena (now LM Arena) is a public platform where anyone can type a prompt and receive responses from two anonymous AI models side-by-side. The user picks a winner. Millions of votes feed an Elo-style rating system. The resulting leaderboard is widely cited as the consumer-preference benchmark · the gap between scores reflects how often humans prefer one model over another on real queries.

Chatbot Arena (LMSYS, UC Berkeley) has collected 15M+ votes since 2023. The rating is Bradley-Terry Elo updated in real time. Models are presented anonymously as "Model A" vs "Model B"; users choose A, B, tie, or both bad. Category filters (coding, reasoning, hard prompts, math) slice the leaderboard by query type. The overall leaderboard is dominated by general-purpose chat quality; specialized categories rank differently. Methodological concerns: voter self-selection bias, short queries dominating, style-over-substance preferences. Despite critiques, Arena is the closest thing to a real-world preference measure in 2026.

The Bradley-Terry model estimates each model's strength θ such that P(A beats B) = 1 / (1 + e^(θ_B - θ_A)). Online updates use stochastic gradient on cross-entropy loss of the win-loss matrix. Confidence intervals via bootstrap resampling · typically ±5-15 Elo at 90% CI. Category leaderboards use the same model but subset prompts by classifier-tagged category. Known biases: verbosity bias (longer responses preferred), markdown/formatting bias, and language-distribution bias. Arena Hard (a curated subset) attempts to control for these with harder prompts.