What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspective
This work provides a statistical understanding of data augmentation's role in knowledge distillation, which is an incremental contribution for researchers and practitioners aiming to improve KD performance.
This paper investigates why certain data augmentation (DA) schemes perform better in knowledge distillation (KD), proposing that effective DA reduces the covariance of teacher-student cross-entropy. They introduce a metric, T. stddev, and a new entropy-based DA scheme, CutMixPick, which enhances CutMix.
Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide the student model. Existing works mainly study KD from the network output side (e.g., trying to design a better KD loss function), while few have attempted to understand it from the input side. Especially, its interplay with data augmentation (DA) has not been well understood. In this paper, we ask: Why do some DA schemes (e.g., CutMix) inherently perform much better than others in KD? What makes a "good" DA in KD? Our investigation from a statistical perspective suggests that a good DA scheme should reduce the covariance of the teacher-student cross-entropy. A practical metric, the stddev of teacher's mean probability (T. stddev), is further presented and well justified empirically. Besides the theoretical understanding, we also introduce a new entropy-based data-mixing DA scheme, CutMixPick, to further enhance CutMix. Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in knowledge distillation.