Influence Functions for Scalable Data Attribution in Diffusion Models
This work addresses data attribution challenges for users of diffusion models, but it is incremental as it builds on existing influence function methods by adapting them to diffusion models.
The authors tackled the problem of data attribution and interpretability in diffusion models by developing an influence functions framework to predict how training data removal affects generated outputs, showing that their method outperforms previous approaches on evaluations like the Linear Data-modelling Score without hyperparameter tuning.
Diffusion models have led to significant advancements in generative modelling. Yet their widespread adoption poses challenges regarding data attribution and interpretability. In this paper, we aim to help address such challenges in diffusion models by developing an influence functions framework. Influence function-based data attribution methods approximate how a model's output would have changed if some training data were removed. In supervised learning, this is usually used for predicting how the loss on a particular example would change. For diffusion models, we focus on predicting the change in the probability of generating a particular example via several proxy measurements. We show how to formulate influence functions for such quantities and how previously proposed methods can be interpreted as particular design choices in our framework. To ensure scalability of the Hessian computations in influence functions, we systematically develop K-FAC approximations based on generalised Gauss-Newton matrices specifically tailored to diffusion models. We recast previously proposed methods as specific design choices in our framework and show that our recommended method outperforms previous data attribution approaches on common evaluations, such as the Linear Data-modelling Score (LDS) or retraining without top influences, without the need for method-specific hyperparameter tuning.