HumanEval

HumanEval was published by OpenAI in 2021 alongside Codex. It contains 164 hand-crafted Python programming problems, each with a function signature, docstring, and several unit tests. The metric is pass@k · the percentage of problems where at least one of k generated solutions passes all tests. Most frontier models now score 90%+ on pass@1, so HumanEval is mostly saturated as a differentiator.

HumanEval's 164 problems span string manipulation, list operations, math, and simple algorithms. The test suite is held out, not shown to the model. Pass@1 is the hardest metric · one shot, must be correct. Pass@10 and pass@100 allow the model multiple attempts. HumanEval+ extends the suite with additional adversarial tests. Frontier models crossed 90% pass@1 in 2023; by 2026 the frontier is 98-99% and the benchmark is used mostly as a sanity check. Real-world coding capability is better measured by SWE-bench, Aider polyglot, or LiveBench coding subset.

Pass@k is estimated from n generated samples using an unbiased estimator 1 - C(n-c, k) / C(n, k) where c is the number of correct samples. Temperature affects this significantly; most reports use temperature 0.2-0.8 and sample 20-100 solutions. Test leakage has been documented · HumanEval problems appear in public training data, limiting its signal. HumanEval+ (2023) adds mutation-based tests to harden the suite. Primary criticism: one-function scope and academic-toy style doesn't predict agent or repo-scale coding performance.