Predictive and Causal Implications of using Shapley Value for Model Interpretation
This work addresses the implications of Shapley value for model interpretation, highlighting its limitations in predictive and causal modeling, which is significant for researchers and practitioners in machine learning seeking reliable explanation tools, but it is incremental as it builds on existing theoretical foundations.
The paper tackles the problem of using Shapley value for model interpretation by analyzing it within a Bayesian network framework, establishing its relationship with conditional independence and showing that high Shapley value does not guarantee predictive importance, while low value can impair performance, and it does not reflect causal relationships.
Shapley value is a concept from game theory. Recently, it has been used for explaining complex models produced by machine learning techniques. Although the mathematical definition of Shapley value is straight-forward, the implication of using it as a model interpretation tool is yet to be described. In the current paper, we analyzed Shapley value in the Bayesian network framework. We established the relationship between Shapley value and conditional independence, a key concept in both predictive and causal modeling. Our results indicate that, eliminating a variable with high Shapley value from a model do not necessarily impair predictive performance, whereas eliminating a variable with low Shapley value from a model could impair performance. Therefore, using Shapley value for feature selection do not result in the most parsimonious and predictively optimal model in the general case. More importantly, Shapley value of a variable do not reflect their causal relationship with the target of interest.