Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation
This work improves dataset distillation for deep learning practitioners by enhancing synthetic dataset quality, though it appears incremental as it builds on existing distribution matching methods.
The paper tackles the problem of dataset distillation by addressing limitations in existing distribution matching methods, specifically dispersed feature distributions within classes and lack of precision in feature consistency, resulting in performance improvements of up to 6.6% on CIFAR10 and robust cross-architecture performance with a maximum drop of 1.7%.
Dataset distillation has emerged as a promising approach in deep learning, enabling efficient training with small synthetic datasets derived from larger real ones. Particularly, distribution matching-based distillation methods attract attention thanks to its effectiveness and low computational cost. However, these methods face two primary limitations: the dispersed feature distribution within the same class in synthetic datasets, reducing class discrimination, and an exclusive focus on mean feature consistency, lacking precision and comprehensiveness. To address these challenges, we introduce two novel constraints: a class centralization constraint and a covariance matching constraint. The class centralization constraint aims to enhance class discrimination by more closely clustering samples within classes. The covariance matching constraint seeks to achieve more accurate feature distribution matching between real and synthetic datasets through local feature covariance matrices, particularly beneficial when sample sizes are much smaller than the number of features. Experiments demonstrate notable improvements with these constraints, yielding performance boosts of up to 6.6% on CIFAR10, 2.9% on SVHN, 2.5% on CIFAR100, and 2.5% on TinyImageNet, compared to the state-of-the-art relevant methods. In addition, our method maintains robust performance in cross-architecture settings, with a maximum performance drop of 1.7% on four architectures. Code is available at https://github.com/VincenDen/IID.