Towards Principled Dataset Distillation: A Spectral Distribution Perspective
This addresses the challenge of efficient model training on imbalanced datasets for machine learning practitioners, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackled the problem of dataset distillation on long-tailed datasets, where existing methods degrade, by proposing Class-Aware Spectral Distribution Matching (CSDM), which achieved a 14.0% improvement over state-of-the-art methods on CIFAR-10-LT with 10 images per class and showed only a 5.7% performance drop when tail class images decreased from 500 to 25.
Dataset distillation (DD) aims to compress large-scale datasets into compact synthetic counterparts for efficient model training. However, existing DD methods exhibit substantial performance degradation on long-tailed datasets. We identify two fundamental challenges: heuristic design choices for distribution discrepancy measure and uniform treatment of imbalanced classes. To address these limitations, we propose Class-Aware Spectral Distribution Matching (CSDM), which reformulates distribution alignment via the spectrum of a well-behaved kernel function. This technique maps the original samples into frequency space, resulting in the Spectral Distribution Distance (SDD). To mitigate class imbalance, we exploit the unified form of SDD to perform amplitude-phase decomposition, which adaptively prioritizes the realism in tail classes. On CIFAR-10-LT, with 10 images per class, CSDM achieves a 14.0% improvement over state-of-the-art DD methods, with only a 5.7% performance drop when the number of images in tail classes decreases from 500 to 25, demonstrating strong stability on long-tailed data.