GRAFT: Gradient-Aware Fast MaxVol Technique for Dynamic Data Sampling
This work addresses the problem of costly neural network training for researchers and practitioners, offering an incremental improvement over existing subset selection methods.
The paper tackles the high computational and environmental costs of training neural networks on large datasets by introducing GRAFT, a dynamic subset selection method that reduces wall-clock time, energy consumption, and CO2 emissions while matching or exceeding baseline accuracy and efficiency across multiple benchmarks.
Training modern neural networks on large datasets is computationally and environmentally costly. We introduce GRAFT, a scalable in-training subset selection method that (i) extracts a low-rank feature representation for each batch, (ii) applies a Fast MaxVol sampler to select a small, diverse subset that spans the batch's dominant subspace, and (iii) dynamically adjusts the subset size using a gradient-approximation criterion. By operating in low-rank subspaces and training on carefully chosen examples instead of full batches, GRAFT preserves the training trajectory while reducing wall-clock time, energy consumption, and $\mathrm{CO}_2$ emissions. Across multiple benchmarks, GRAFT matches or exceeds recent selection baselines in both accuracy and efficiency, providing a favorable trade-off between accuracy, efficiency, and emissions.