Data-to-Model Distillation: Data-Efficient Learning Framework
This addresses computational inefficiency and scalability issues in dataset distillation for machine learning practitioners, offering a more flexible and generalizable approach.
The paper tackles the problem of dataset distillation by proposing Data-to-Model Distillation (D2M), which distills knowledge from a large dataset into a pre-trained generative model to efficiently produce synthetic training data, achieving superior performance and scalability up to 128x128 ImageNet-1K across 15 datasets.
Dataset distillation aims to distill the knowledge of a large-scale real dataset into small yet informative synthetic data such that a model trained on it performs as well as a model trained on the full dataset. Despite recent progress, existing dataset distillation methods often struggle with computational efficiency, scalability to complex high-resolution datasets, and generalizability to deep architectures. These approaches typically require retraining when the distillation ratio changes, as knowledge is embedded in raw pixels. In this paper, we propose a novel framework called Data-to-Model Distillation (D2M) to distill the real dataset's knowledge into the learnable parameters of a pre-trained generative model by aligning rich representations extracted from real and generated images. The learned generative model can then produce informative training images for different distillation ratios and deep architectures. Extensive experiments on 15 datasets of varying resolutions show D2M's superior performance, re-distillation efficiency, and cross-architecture generalizability. Our method effectively scales up to high-resolution 128x128 ImageNet-1K. Furthermore, we verify D2M's practical benefits for downstream applications in neural architecture search.