Manifold Restricted Interventional Shapley Values
This addresses the issue of unreliable explanations in machine learning models for practitioners relying on interpretability tools, though it is an incremental improvement over existing Shapley methods.
The paper tackles the problem that existing Shapley value methods for explaining model predictions are either sensitive to out-of-distribution inputs or overly dependent on input data, leading to misleading explanations. It proposes ManifoldShap, which restricts evaluations to the data manifold, and shows it is robust to off-manifold perturbations and leads to more accurate and intuitive explanations than state-of-the-art methods.
Shapley values are model-agnostic methods for explaining model predictions. Many commonly used methods of computing Shapley values, known as off-manifold methods, rely on model evaluations on out-of-distribution input samples. Consequently, explanations obtained are sensitive to model behaviour outside the data distribution, which may be irrelevant for all practical purposes. While on-manifold methods have been proposed which do not suffer from this problem, we show that such methods are overly dependent on the input data distribution, and therefore result in unintuitive and misleading explanations. To circumvent these problems, we propose ManifoldShap, which respects the model's domain of validity by restricting model evaluations to the data manifold. We show, theoretically and empirically, that ManifoldShap is robust to off-manifold perturbations of the model and leads to more accurate and intuitive explanations than existing state-of-the-art Shapley methods.