Uncertainty-based Debiasing and Unlearning for Decontamination
This work addresses the problem of data contamination in LLM benchmarks, offering a practical decontamination method that does not require an uncontaminated reference model.
The authors propose a sample-level evaluation framework for decontamination of LLMs using distributional distance metrics, and introduce Uncertainty-Based Decontamination (UBD) methods that estimate per-sample memorization via deep ensembles without needing an uncontaminated model. UBD achieves output distributions closer to uncontaminated models than baselines on MMLU-Pro and MATH-MCQA while preserving performance on clean data.
Benchmark-based evaluation is the dominant paradigm for assessing large language model (LLM) capabilities, yet data contamination inflates reported performance and undermines fair comparison. Existing decontamination methods are evaluated solely through aggregate accuracy, which can obscure substantial differences in per-sample model behaviour, and many require access to an uncontaminated model. In this paper, we propose a sample-level evaluation framework for decontamination that complements accuracy-based assessment with distributional distance metrics, measuring how closely a decontaminated model recovers the output distribution of an uncontaminated model on each sample. Building on this framework, we introduce Uncertainty-Based Decontamination (UBD), a family of methods that leverage deep ensembles of the contaminated model to estimate per-sample memorization without requiring a uncontaminated model or knowledge of which samples are contaminated. UBD estimates a per-sample correction scalar from ensemble uncertainty, which is used to construct a debiased target distribution that suppresses the inflated probability mass on correct answers induced by contamination. This target is then used either as a post-hoc output correction (debiasing) or as a soft training signal for parameter update (unlearning). Experiments on MMLU-Pro and MATH-MCQA across multiple LLM backbones demonstrate that UBD produces per-sample output distributions substantially closer to those of an uncontaminated model than paraphrasing or choice-permutation baselines, while preserving model performance on uncontaminated data.