Data Distillation: A Survey
It addresses efficiency issues for researchers and practitioners in machine learning, but is incremental as it synthesizes existing work without introducing new methods.
This survey tackles the problem of training large deep learning models on massive datasets, which causes high training time, slow research iteration, and poor eco-sustainability, by reviewing data distillation methods that synthesize concise data summaries as drop-in replacements for tasks like model training and inference.
The popularity of deep learning has led to the curation of a vast number of massive and multifarious datasets. Despite having close-to-human performance on individual tasks, training parameter-hungry models on large datasets poses multi-faceted problems such as (a) high model-training time; (b) slow research iteration; and (c) poor eco-sustainability. As an alternative, data distillation approaches aim to synthesize terse data summaries, which can serve as effective drop-in replacements of the original dataset for scenarios like model training, inference, architecture search, etc. In this survey, we present a formal framework for data distillation, along with providing a detailed taxonomy of existing approaches. Additionally, we cover data distillation approaches for different data modalities, namely images, graphs, and user-item interactions (recommender systems), while also identifying current challenges and future research directions.