MAGIC: Near-Optimal Data Attribution for Deep Learning
This addresses the challenge of understanding data influence in deep learning for researchers and practitioners, though it appears incremental as it builds on prior methods.
The paper tackles the problem of predictive data attribution in large-scale non-convex deep learning settings, where existing methods perform poorly, and presents MAGIC, a new method that combines classical approaches with metadifferentiation to achieve near-optimal estimates of how training data changes affect model predictions.
The goal of predictive data attribution is to estimate how adding or removing a given set of training datapoints will affect model predictions. In convex settings, this goal is straightforward (i.e., via the infinitesimal jackknife). In large-scale (non-convex) settings, however, existing methods are far less successful -- current methods' estimates often only weakly correlate with ground truth. In this work, we present a new data attribution method (MAGIC) that combines classical methods and recent advances in metadifferentiation to (nearly) optimally estimate the effect of adding or removing training data on model predictions.