Data pruning and neural scaling laws: fundamental limitations of score-based algorithms
This addresses the need for effective data pruning to enhance neural scaling laws, but the findings are incremental as they focus on specific algorithm types.
The paper demonstrates that score-based data pruning algorithms fail in the high compression regime (keeping 30% or less of data), showing theoretical and empirical limitations, and proposes calibration protocols with randomization to improve performance.
Data pruning algorithms are commonly used to reduce the memory and computational cost of the optimization process. Recent empirical results reveal that random data pruning remains a strong baseline and outperforms most existing data pruning methods in the high compression regime, i.e., where a fraction of $30\%$ or less of the data is kept. This regime has recently attracted a lot of interest as a result of the role of data pruning in improving the so-called neural scaling laws; in [Sorscher et al.], the authors showed the need for high-quality data pruning algorithms in order to beat the sample power law. In this work, we focus on score-based data pruning algorithms and show theoretically and empirically why such algorithms fail in the high compression regime. We demonstrate ``No Free Lunch" theorems for data pruning and present calibration protocols that enhance the performance of existing pruning algorithms in this high compression regime using randomization.