Dataset Distillation by Automatic Training Trajectories
This addresses a specific bottleneck in dataset distillation for machine learning practitioners, offering an incremental improvement over existing methods.
The paper tackles the Accumulated Mismatching Problem in dataset distillation, where fixed trajectory lengths cause overfitting and reduce generality across architectures, by proposing Automatic Training Trajectories that dynamically adjusts trajectory length, resulting in improved performance in cross-architecture tests and enhanced stability.
Dataset Distillation is used to create a concise, yet informative, synthetic dataset that can replace the original dataset for training purposes. Some leading methods in this domain prioritize long-range matching, involving the unrolling of training trajectories with a fixed number of steps (NS) on the synthetic dataset to align with various expert training trajectories. However, traditional long-range matching methods possess an overfitting-like problem, the fixed step size NS forces synthetic dataset to distortedly conform seen expert training trajectories, resulting in a loss of generality-especially to those from unencountered architecture. We refer to this as the Accumulated Mismatching Problem (AMP), and propose a new approach, Automatic Training Trajectories (ATT), which dynamically and adaptively adjusts trajectory length NS to address the AMP. Our method outperforms existing methods particularly in tests involving cross-architectures. Moreover, owing to its adaptive nature, it exhibits enhanced stability in the face of parameter variations.