AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation
This is an incremental improvement for machine learning practitioners working on knowledge distillation, as it enhances existing methods by aligning predictive distributions more effectively.
The paper tackles the problem of improving self-knowledge distillation by regularizing training with adversarial learning and implicit distillations, resulting in better performance than state-of-the-art methods on multiple datasets.
We present a novel adversarial penalized self-knowledge distillation method, named adversarial learning and implicit regularization for self-knowledge distillation (AI-KD), which regularizes the training procedure by adversarial learning and implicit distillations. Our model not only distills the deterministic and progressive knowledge which are from the pre-trained and previous epoch predictive probabilities but also transfers the knowledge of the deterministic predictive distributions using adversarial learning. The motivation is that the self-knowledge distillation methods regularize the predictive probabilities with soft targets, but the exact distributions may be hard to predict. Our method deploys a discriminator to distinguish the distributions between the pre-trained and student models while the student model is trained to fool the discriminator in the trained procedure. Thus, the student model not only can learn the pre-trained model's predictive probabilities but also align the distributions between the pre-trained and student models. We demonstrate the effectiveness of the proposed method with network architectures on multiple datasets and show the proposed method achieves better performance than state-of-the-art methods.