Dataset Distillation with Convexified Implicit Gradients
This work addresses the problem of efficiently distilling large datasets into smaller synthetic sets for machine learning practitioners, representing a strong incremental advance in the field.
The paper tackled dataset distillation by proposing a new algorithm using reparameterization and convexified implicit gradients (RCIG), which achieved substantial improvements over the state-of-the-art, such as a 108% average gain on resized ImageNet with one image per class.
We propose a new dataset distillation algorithm using reparameterization and convexification of implicit gradients (RCIG), that substantially improves the state-of-the-art. To this end, we first formulate dataset distillation as a bi-level optimization problem. Then, we show how implicit gradients can be effectively used to compute meta-gradient updates. We further equip the algorithm with a convexified approximation that corresponds to learning on top of a frozen finite-width neural tangent kernel. Finally, we improve bias in implicit gradients by parameterizing the neural network to enable analytical computation of final-layer parameters given the body parameters. RCIG establishes the new state-of-the-art on a diverse series of dataset distillation tasks. Notably, with one image per class, on resized ImageNet, RCIG sees on average a 108\% improvement over the previous state-of-the-art distillation algorithm. Similarly, we observed a 66\% gain over SOTA on Tiny-ImageNet and 37\% on CIFAR-100.