Glossary
Benchmark saturation
When top models score so close to the maximum on a test that it no longer separates them.
A benchmark is saturated when leading models score near the top of its range. At that point small differences in score mostly reflect noise, errors in the test itself, or details of how it was run, rather than real differences in ability.
Saturation is why the AI field keeps producing harder benchmarks. It also means that a "new record" on an old benchmark may say little.
Some remaining errors on a saturated benchmark come from questions with wrong or disputed answer keys, which no model can be expected to get "right."
Read more: our explainer