Towards Robust Influence Functions with Flat Validation Minima
This work addresses a specific issue in model interpretability for researchers and practitioners using influence functions, but it is incremental as it builds on prior methods by focusing on validation risk sharpness.
The paper tackled the problem of unreliable influence estimates in deep neural networks with noisy data by identifying deficiencies in loss change estimation due to sharp validation risk, and introduced a novel influence function form for flat validation minima, showing experimental superiority across tasks.
The Influence Function (IF) is a widely used technique for assessing the impact of individual training samples on model predictions. However, existing IF methods often fail to provide reliable influence estimates in deep neural networks, particularly when applied to noisy training data. This issue does not stem from inaccuracies in parameter change estimation, which has been the primary focus of prior research, but rather from deficiencies in loss change estimation, specifically due to the sharpness of validation risk. In this work, we establish a theoretical connection between influence estimation error, validation set risk, and its sharpness, underscoring the importance of flat validation minima for accurate influence estimation. Furthermore, we introduce a novel estimation form of Influence Function specifically designed for flat validation minima. Experimental results across various tasks validate the superiority of our approach.