Explaining Probabilistic Models with Distributional Values
This work addresses interpretability issues in explainable AI for probabilistic models, offering a novel framework that provides more fine-grained explanations, though it is incremental in building on cooperative game theory.
The paper tackles the mismatch between what probabilistic models output and what existing explanation methods like SHAP explain, by introducing distributional values that track changes in model outputs such as class flips, and demonstrates their effectiveness with case studies on vision and language models.
A large branch of explainable machine learning is grounded in cooperative game theory. However, research indicates that game-theoretic explanations may mislead or be hard to interpret. We argue that often there is a critical mismatch between what one wishes to explain (e.g. the output of a classifier) and what current methods such as SHAP explain (e.g. the scalar probability of a class). This paper addresses such gap for probabilistic models by generalising cooperative games and value operators. We introduce the distributional values, random variables that track changes in the model output (e.g. flipping of the predicted class) and derive their analytic expressions for games with Gaussian, Bernoulli and Categorical payoffs. We further establish several characterising properties, and show that our framework provides fine-grained and insightful explanations with case studies on vision and language models.