Accumulative SGD Influence Estimation for Data Attribution
This addresses the need for precise data attribution in data-centric AI, though it appears to be an incremental improvement over existing influence estimation methods.
The paper tackles the problem of inaccurate per-sample influence estimation in SGD training by proposing ACC-SGD-IE, a trajectory-aware estimator that accounts for cross-epoch compounding effects. The method yields more accurate influence estimates across various datasets and training regimes, with downstream data cleansing producing models that outperform those using standard SGD-IE.
Modern data-centric AI needs precise per-sample influence. Standard SGD-IE approximates leave-one-out effects by summing per-epoch surrogates and ignores cross-epoch compounding, which misranks critical examples. We propose ACC-SGD-IE, a trajectory-aware estimator that propagates the leave-one-out perturbation across training and updates an accumulative influence state at each step. In smooth strongly convex settings it achieves geometric error contraction and, in smooth non-convex regimes, it tightens error bounds; larger mini-batches further reduce constants. Empirically, on Adult, 20 Newsgroups, and MNIST under clean and corrupted data and both convex and non-convex training, ACC-SGD-IE yields more accurate influence estimates, especially over long epochs. For downstream data cleansing it more reliably flags noisy samples, producing models trained on ACC-SGD-IE cleaned data that outperform those cleaned with SGD-IE.