Glossary
Benchmark contamination
When test questions, or their answers, have leaked into a model's training data, inflating its score.
Contamination happens when the questions or answers from a benchmark appear in the data a model was trained on. The model may then score well because it has effectively seen the test, not because it can solve new problems of that kind.
Because training data is drawn from the public web, popular benchmarks are easily contaminated, often by accident. Researchers try to detect it, keep test sets private, or build new tests from material created after a model's training cutoff.
A large jump on an old, widely published benchmark deserves more caution than a strong result on a fresh or held-out test.
Read more: our explainer