Leveraging Unlabeled Data to Track Memorization
This addresses the issue of noisy label memorization for machine learning practitioners, offering a practical tool for model evaluation, though it is incremental as it builds on existing memorization tracking methods.
The authors tackled the problem of deep neural networks memorizing noisy labels, which harms generalization, by proposing a metric called susceptibility to track memorization without needing ground-truth labels. They demonstrated its effectiveness across architectures and datasets, showing it helps identify models that generalize well to clean data.
Deep neural networks may easily memorize noisy labels present in real-world data, which degrades their ability to generalize. It is therefore important to track and evaluate the robustness of models against noisy label memorization. We propose a metric, called susceptibility, to gauge such memorization for neural networks. Susceptibility is simple and easy to compute during training. Moreover, it does not require access to ground-truth labels and it only uses unlabeled data. We empirically show the effectiveness of our metric in tracking memorization on various architectures and datasets and provide theoretical insights into the design of the susceptibility metric. Finally, we show through extensive experiments on datasets with synthetic and real-world label noise that one can utilize susceptibility and the overall training accuracy to distinguish models that maintain a low memorization on the training set and generalize well to unseen clean data.