Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation
This work addresses data storage and communication challenges in scaling neural networks by improving dataset distillation, though it appears incremental as it builds on existing coreset and generative model techniques.
The paper tackles the computational efficiency and quality limitations of dataset distillation methods by proposing a two-stage approach that first compresses datasets into informative coresets and then uses generative foundation models to dynamically expand them with enhanced resolution and variability. The method achieves over 10% improvement compared to state-of-the-art on several large-scale benchmarks.
With the rapid scaling of neural networks, data storage and communication demands have intensified. Dataset distillation has emerged as a promising solution, condensing information from extensive datasets into a compact set of synthetic samples by solving a bilevel optimization problem. However, current methods face challenges in computational efficiency, particularly with high-resolution data and complex architectures. Recently, knowledge-distillation-based dataset condensation approaches have made this process more computationally feasible. Yet, with the recent developments of generative foundation models, there is now an opportunity to achieve even greater compression, enhance the quality of distilled data, and introduce valuable diversity into the data representation. In this work, we propose a two-stage solution. First, we compress the dataset by selecting only the most informative patches to form a coreset. Next, we leverage a generative foundation model to dynamically expand this compressed set in real-time, enhancing the resolution of these patches and introducing controlled variability to the coreset. Our extensive experiments demonstrate the robustness and efficiency of our approach across a range of dataset distillation benchmarks. We demonstrate a significant improvement of over 10% compared to the state-of-the-art on several large-scale dataset distillation benchmarks. The code will be released soon.