From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
This work addresses the need for interpretability in machine learning, particularly for practitioners, by proposing a novel framework that bridges human explanation concepts with model interpretability, though it is incremental in adapting existing concepts to new settings.
The paper tackles the problem of making machine learning models interpretable by developing a framework based on weight of evidence from information theory, which generates explanations that are meaningful to humans and are shown to be accurate, robust, and usable in a user study with practitioners.
We take inspiration from the study of human explanation to inform the design and evaluation of interpretability methods in machine learning. First, we survey the literature on human explanation in philosophy, cognitive science, and the social sciences, and propose a list of design principles for machine-generated explanations that are meaningful to humans. Using the concept of weight of evidence from information theory, we develop a method for generating explanations that adhere to these principles. We show that this method can be adapted to handle high-dimensional, multi-class settings, yielding a flexible framework for generating explanations. We demonstrate that these explanations can be estimated accurately from finite samples and are robust to small perturbations of the inputs. We also evaluate our method through a qualitative user study with machine learning practitioners, where we observe that the resulting explanations are usable despite some participants struggling with background concepts like prior class probabilities. Finally, we conclude by surfacing~design~implications for interpretability tools in general.