What is benchmark contamination in model evaluation?

Benchmark contamination occurs when evaluation data leaks into training data.

Contamination happens when models are trained on data that overlaps with evaluation benchmarks, intentionally or unintentionally. This leads to inflated performance and misleading comparisons. It is a major issue in large language model evaluation where web-scale datasets may include benchmark questions.