The State of Knowledge Distillation for Classification
This work addresses reproducibility and generalizability issues in knowledge distillation for classification, which is incremental as it builds on existing methods.
The paper investigates knowledge distillation strategies for classification tasks, finding that many state-of-the-art methods are hard to reproduce and lack generalizability, while a tuned classical distillation with data augmentation provides orthogonal improvements.
We survey various knowledge distillation (KD) strategies for simple classification tasks and implement a set of techniques that claim state-of-the-art accuracy. Our experiments using standardized model architectures, fixed compute budgets, and consistent training schedules indicate that many of these distillation results are hard to reproduce. This is especially apparent with methods using some form of feature distillation. Further examination reveals a lack of generalizability where these techniques may only succeed for specific architectures and training settings. We observe that appropriately tuned classical distillation in combination with a data augmentation training scheme gives an orthogonal improvement over other techniques. We validate this approach and open-source our code.