Generalized Kernel Inducing Points by Duality Gap for Dataset Distillation
This work addresses a bottleneck in dataset distillation for machine learning practitioners by extending applicability to classification tasks, though it is incremental as it builds directly on the KIP method.
The paper tackles the limitation of Kernel Inducing Points (KIP) in dataset distillation, which was restricted to squared loss, by proposing Duality Gap KIP (DGKIP) that eliminates bi-level optimization and supports loss functions like cross-entropy and hinge loss, with experimental results on MNIST and CIFAR-10 showing retained efficiency and robust performance.
We propose Duality Gap KIP (DGKIP), an extension of the Kernel Inducing Points (KIP) method for dataset distillation. While existing dataset distillation methods often rely on bi-level optimization, DGKIP eliminates the need for such optimization by leveraging duality theory in convex programming. The KIP method has been introduced as a way to avoid bi-level optimization; however, it is limited to the squared loss and does not support other loss functions (e.g., cross-entropy or hinge loss) that are more suitable for classification tasks. DGKIP addresses this limitation by exploiting an upper bound on parameter changes after dataset distillation using the duality gap, enabling its application to a wider range of loss functions. We also characterize theoretical properties of DGKIP by providing upper bounds on the test error and prediction consistency after dataset distillation. Experimental results on standard benchmarks such as MNIST and CIFAR-10 demonstrate that DGKIP retains the efficiency of KIP while offering broader applicability and robust performance.