Pseudo-label Correction for Instance-dependent Noise Using Teacher-student Framework
This addresses the challenge of poor generalization due to label noise in deep learning models, which is an incremental improvement over existing methods.
The paper tackles the problem of label noise in deep learning by proposing a teacher-student framework called P-LC for pseudo-label correction, achieving superior performance over state-of-the-art methods on datasets like MNIST, Fashion-MNIST, and SVHN, especially at high noise levels.
The high capacity of deep learning models to learn complex patterns poses a significant challenge when confronted with label noise. The inability to differentiate clean and noisy labels ultimately results in poor generalization. We approach this problem by reassigning the label for each image using a new teacher-student based framework termed P-LC (pseudo-label correction). Traditional teacher-student networks are composed of teacher and student classifiers for knowledge distillation. In our novel approach, we reconfigure the teacher network into a triple encoder, leveraging the triplet loss to establish a pseudo-label correction system. As the student generates pseudo labels for a set of given images, the teacher learns to choose between the initially assigned labels and the pseudo labels. Experiments on MNIST, Fashion-MNIST, and SVHN demonstrate P-LC's superior performance over existing state-of-the-art methods across all noise levels, most notably in high noise. In addition, we introduce a noise level estimation to help assess model performance and inform the need for additional data cleaning procedures.