What Makes a Good Dataset for Knowledge Distillation?
This addresses a practical issue for practitioners in model compression, especially in scenarios like continual learning or using proprietary data, but it is incremental as it builds on existing knowledge distillation methods.
The paper tackles the problem of knowledge distillation when the teacher's original dataset is unavailable, exploring various surrogate datasets and demonstrating that even synthetic imagery can be effective alternatives, identifying criteria for what makes a good distillation dataset.
Knowledge distillation (KD) has been a popular and effective method for model compression. One important assumption of KD is that the teacher's original dataset will also be available when training the student. However, in situations such as continual learning and distilling large models trained on company-withheld datasets, having access to the original data may not always be possible. This leads practitioners towards utilizing other sources of supplemental data, which could yield mixed results. One must then ask: "what makes a good dataset for transferring knowledge from teacher to student?" Many would assume that only real in-domain imagery is viable, but is that the only option? In this work, we explore multiple possible surrogate distillation datasets and demonstrate that many different datasets, even unnatural synthetic imagery, can serve as a suitable alternative in KD. From examining these alternative datasets, we identify and present various criteria describing what makes a good dataset for distillation. Source code is available at https://github.com/osu-cvl/good-kd-dataset.