On Revisiting Entropy for Identifying Mislabeled Images
This work provides a method for improving the robustness of deep learning models for medical imaging by identifying and mitigating the impact of mislabeled data, a common issue in this domain.
This paper addresses the problem of mislabeled images in training datasets, which degrades deep network performance. The authors propose a signed entropy integral (SEI) statistic that leverages training dynamics to identify mislabeled samples, achieving state-of-the-art performance on four medical imaging datasets.
Mislabeled samples in training datasets severely degrade the performance of deep networks, as overparameterized models tend to memorize erroneous labels. We address this challenge by proposing a novel approach for mislabeled data detection that leverages training dynamics. Our method is grounded in the key observation that correctly labeled samples exhibit consistent entropy decrease during training, while mislabeled samples maintain relatively high entropy throughout the training process. Building on this insight, we introduce a signed entropy integral (SEI) statistic that captures both the magnitude and temporal trend of prediction entropy across training epochs. SEI is broadly applicable to classification networks and demonstrates particular effectiveness when integrated with contrastive language-image pretraining (CLIP) architectures. Through extensive experiments on four medical imaging datasets -- a domain particularly susceptible to labeling errors due to diagnostic complexity -- spanning diverse modalities and pathologies, we demonstrate that SEI achieves state-of-the-art performance in mislabeled data identification, outperforming existing methods while maintaining computational efficiency and implementation simplicity. Our code is available at https://github.com/MedAITech/SEI.