Accelerating Dataset Distillation via Model Augmentation
This work addresses the efficiency problem in dataset distillation for machine learning practitioners, representing an incremental improvement over existing gradient matching approaches.
The paper tackles the computational intensity of dataset distillation methods by proposing model augmentation techniques using early-stage models and parameter perturbation, achieving up to 20x speedup while maintaining performance comparable to state-of-the-art methods.
Dataset Distillation (DD), a newly emerging field, aims at generating much smaller but efficient synthetic training datasets from large ones. Existing DD methods based on gradient matching achieve leading performance; however, they are extremely computationally intensive as they require continuously optimizing a dataset among thousands of randomly initialized models. In this paper, we assume that training the synthetic data with diverse models leads to better generalization performance. Thus we propose two model augmentation techniques, i.e. using early-stage models and parameter perturbation to learn an informative synthetic set with significantly reduced training cost. Extensive experiments demonstrate that our method achieves up to 20x speedup and comparable performance on par with state-of-the-art methods.