What is semantic deduplication in evaluation datasets?

Semantic deduplication removes meaning-similar duplicates in datasets.

Unlike exact matching, semantic deduplication uses embeddings to detect paraphrased or semantically similar samples. This prevents inflated evaluation scores and ensures diversity in benchmarks. It is critical in LLM evaluation pipelines.