Toward Efficient Influence Function: Dropout as a Compression Tool
This addresses a bottleneck in model transparency and data selection for machine learning practitioners, though it is an incremental improvement on existing approximation methods.
The paper tackles the computational and memory inefficiency of influence functions for large-scale models by using dropout as a gradient compression tool, significantly reducing overhead while preserving data influence.
Assessing the impact the training data on machine learning models is crucial for understanding the behavior of the model, enhancing the transparency, and selecting training data. Influence function provides a theoretical framework for quantifying the effect of training data points on model's performance given a specific test data. However, the computational and memory costs of influence function presents significant challenges, especially for large-scale models, even when using approximation methods, since the gradients involved in computation are as large as the model itself. In this work, we introduce a novel approach that leverages dropout as a gradient compression mechanism to compute the influence function more efficiently. Our method significantly reduces computational and memory overhead, not only during the influence function computation but also in gradient compression process. Through theoretical analysis and empirical validation, we demonstrate that our method could preserves critical components of the data influence and enables its application to modern large-scale models.