Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks
This addresses the challenge of dataset storage and retraining efficiency for machine learning practitioners, offering a novel approach with broad applications in continual learning and dataset compression.
The paper tackles the problem of compressing large datasets into compact, addressable memories to enable efficient neural network retraining without storing the full dataset, achieving state-of-the-art results with up to 16.5% and 9.7% retained accuracy improvements on CIFAR10 and CIFAR100, respectively, and 23.2% accuracy improvement in continual learning on MANY.
We propose an algorithm that compresses the critical information of a large dataset into compact addressable memories. These memories can then be recalled to quickly re-train a neural network and recover the performance (instead of storing and re-training on the full original dataset). Building upon the dataset distillation framework, we make a key observation that a shared common representation allows for more efficient and effective distillation. Concretely, we learn a set of bases (aka ``memories'') which are shared between classes and combined through learned flexible addressing functions to generate a diverse set of training examples. This leads to several benefits: 1) the size of compressed data does not necessarily grow linearly with the number of classes; 2) an overall higher compression rate with more effective distillation is achieved; and 3) more generalized queries are allowed beyond recalling the original classes. We demonstrate state-of-the-art results on the dataset distillation task across six benchmarks, including up to 16.5% and 9.7% in retained accuracy improvement when distilling CIFAR10 and CIFAR100 respectively. We then leverage our framework to perform continual learning, achieving state-of-the-art results on four benchmarks, with 23.2% accuracy improvement on MANY. The code is released on our project webpage https://github.com/princetonvisualai/RememberThePast-DatasetDistillation.