Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning
This work addresses the challenge of robust training for deep learning models in noisy or adversarial settings, offering a unified approach that benefits both supervised and self-supervised learning, though it appears incremental by building on existing training dynamics analysis.
The paper tackles the problem of training deep neural networks on corrupted data, such as with label noise or adversarial examples, by proposing self-adaptive training, which dynamically uses model predictions to improve generalization without extra computational cost. Experiments on datasets like CIFAR and ImageNet show effectiveness in applications like classification with label noise and self-supervised learning.
We propose self-adaptive training -- a unified training algorithm that dynamically calibrates and enhances training processes by model predictions without incurring an extra computational cost -- to advance both supervised and self-supervised learning of deep neural networks. We analyze the training dynamics of deep networks on training data that are corrupted by, e.g., random noise and adversarial examples. Our analysis shows that model predictions are able to magnify useful underlying information in data and this phenomenon occurs broadly even in the absence of any label information, highlighting that model predictions could substantially benefit the training processes: self-adaptive training improves the generalization of deep networks under noise and enhances the self-supervised representation learning. The analysis also sheds light on understanding deep learning, e.g., a potential explanation of the recently-discovered double-descent phenomenon in empirical risk minimization and the collapsing issue of the state-of-the-art self-supervised learning algorithms. Experiments on the CIFAR, STL, and ImageNet datasets verify the effectiveness of our approach in three applications: classification with label noise, selective classification, and linear evaluation. To facilitate future research, the code has been made publicly available at https://github.com/LayneH/self-adaptive-training.