On Influence Functions, Classification Influence, Relative Influence, Memorization and Generalization
This work addresses efficiency and interpretability challenges for practitioners in large-scale ML, though it is incremental as it builds on existing influence function theory.
The paper tackles the computational burden of influence functions in large-scale ML systems by proposing methods to simplify their calculations and demonstrates that influence sign can distinguish memorization from generalization in training points.
Machine learning systems such as large scale recommendation systems or natural language processing systems are usually trained on billions of training points and are associated with hundreds of billions or trillions of parameters. Improving the learning process in such a way that both the training load is reduced and the model accuracy improved is highly desired. In this paper we take a first step toward solving this problem, studying influence functions from the perspective of simplifying the computations they involve. We discuss assumptions, under which influence computations can be performed on significantly fewer parameters. We also demonstrate that the sign of the influence value can indicate whether a training point is to memorize, as opposed to generalize upon. For this purpose we formally define what memorization means for a training point, as opposed to generalization. We conclude that influence functions can be made practical, even for large scale machine learning systems, and that influence values can be taken into account by algorithms that selectively remove training points, as part of the learning process.