Blind Knowledge Distillation for Robust Image Classification
This addresses the challenge of robust image classification for real-world applications with noisy data, representing an incremental improvement over existing methods.
The paper tackles the problem of training neural networks with noisy labels by introducing Blind Knowledge Distillation, which masks teacher outputs to filter corrupted knowledge and estimate overfitting, resulting in improved detection of clean and noisy labels on the CIFAR-N dataset.
Optimizing neural networks with noisy labels is a challenging task, especially if the label set contains real-world noise. Networks tend to generalize to reasonable patterns in the early training stages and overfit to specific details of noisy samples in the latter ones. We introduce Blind Knowledge Distillation - a novel teacher-student approach for learning with noisy labels by masking the ground truth related teacher output to filter out potentially corrupted knowledge and to estimate the tipping point from generalizing to overfitting. Based on this, we enable the estimation of noise in the training data with Otsus algorithm. With this estimation, we train the network with a modified weighted cross-entropy loss function. We show in our experiments that Blind Knowledge Distillation detects overfitting effectively during training and improves the detection of clean and noisy labels on the recently published CIFAR-N dataset. Code is available at GitHub.