Dataset Distillation: A Comprehensive Review
It addresses the problem of data inefficiency and privacy concerns in deep learning for researchers and practitioners, but is incremental as it reviews existing methods rather than introducing new ones.
This paper provides a comprehensive review of dataset distillation (DD), a technique that aims to create a much smaller synthetic dataset from an original dataset, enabling trained models to achieve performance comparable to those trained on the full dataset, thereby addressing issues like storage, transmission, and privacy.
Recent success of deep learning is largely attributed to the sheer amount of data used for training deep neural networks.Despite the unprecedented success, the massive data, unfortunately, significantly increases the burden on storage and transmission and further gives rise to a cumbersome model training process. Besides, relying on the raw data for training \emph{per se} yields concerns about privacy and copyright. To alleviate these shortcomings, dataset distillation~(DD), also known as dataset condensation (DC), was introduced and has recently attracted much research attention in the community. Given an original dataset, DD aims to derive a much smaller dataset containing synthetic samples, based on which the trained models yield performance comparable with those trained on the original dataset. In this paper, we give a comprehensive review and summary of recent advances in DD and its application. We first introduce the task formally and propose an overall algorithmic framework followed by all existing DD methods. Next, we provide a systematic taxonomy of current methodologies in this area, and discuss their theoretical interconnections. We also present current challenges in DD through extensive experiments and envision possible directions for future works.