Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillation
This work addresses dataset distillation for machine learning practitioners by providing a theoretical understanding and a plug-in method to boost existing techniques, though it is incremental as it builds on prior matching-based approaches.
The paper tackles the problem of dataset distillation by analyzing sample difficulty and finds that prioritizing easier samples improves distilled dataset quality, especially with low image-per-class counts, leading to enhanced performance across multiple methods and datasets.
Dataset Distillation (DD) aims to synthesize a small dataset capable of performing comparably to the original dataset. Despite the success of numerous DD methods, theoretical exploration of this area remains unaddressed. In this paper, we take an initial step towards understanding various matching-based DD methods from the perspective of sample difficulty. We begin by empirically examining sample difficulty, measured by gradient norm, and observe that different matching-based methods roughly correspond to specific difficulty tendencies. We then extend the neural scaling laws of data pruning to DD to theoretically explain these matching-based methods. Our findings suggest that prioritizing the synthesis of easier samples from the original dataset can enhance the quality of distilled datasets, especially in low IPC (image-per-class) settings. Based on our empirical observations and theoretical analysis, we introduce the Sample Difficulty Correction (SDC) approach, designed to predominantly generate easier samples to achieve higher dataset quality. Our SDC can be seamlessly integrated into existing methods as a plugin with minimal code adjustments. Experimental results demonstrate that adding SDC generates higher-quality distilled datasets across 7 distillation methods and 6 datasets.