Mixed Sample Augmentation for Online Distillation
This work addresses the challenge of enhancing online distillation for deep learning practitioners, offering incremental improvements by adapting MSR to a new context.
The paper tackles the problem of improving online knowledge distillation by incorporating Mixed Sample Regularization (MSR), proposing a new method called Cut^nMix and a framework with feature-level mutual learning and a self-ensemble teacher. The result is consistent outperformance of state-of-the-art distillation methods on CIFAR10 and CIFAR100 across six network architectures.
Mixed Sample Regularization (MSR), such as MixUp or CutMix, is a powerful data augmentation strategy to generalize convolutional neural networks. Previous empirical analysis has illustrated an orthogonal performance gain between MSR and conventional offline Knowledge Distillation (KD). To be more specific, student networks can be enhanced with the involvement of MSR in the training stage of sequential distillation. Yet, the interplay between MSR and online knowledge distillation, where an ensemble of peer students learn mutually from each other, remains unexplored. To bridge the gap, we make the first attempt at incorporating CutMix into online distillation, where we empirically observe a significant improvement. Encouraged by this fact, we propose an even stronger MSR specifically for online distillation, named as Cut\textsuperscript{n}Mix. Furthermore, a novel online distillation framework is designed upon Cut\textsuperscript{n}Mix, to enhance the distillation with feature level mutual learning and a self-ensemble teacher. Comprehensive evaluations on CIFAR10 and CIFAR100 with six network architectures show that our approach can consistently outperform state-of-the-art distillation methods.