Interpreting Black Box Models via Hypothesis Testing
This addresses the need for reliable interpretations in high-stakes fields like science and medicine, offering a novel approach to reduce false discoveries.
The paper tackles the problem of false discoveries in black box model interpretations by reframing interpretability as a multiple hypothesis testing problem, proposing two methods that control error rates and show high power in simulations and intuitive feature selections in vision and language models.
In science and medicine, model interpretations may be reported as discoveries of natural phenomena or used to guide patient treatments. In such high-stakes tasks, false discoveries may lead investigators astray. These applications would therefore benefit from control over the finite-sample error rate of interpretations. We reframe black box model interpretability as a multiple hypothesis testing problem. The task is to discover "important" features by testing whether the model prediction is significantly different from what would be expected if the features were replaced with uninformative counterfactuals. We propose two testing methods: one that provably controls the false discovery rate but which is not yet feasible for large-scale applications, and an approximate testing method which can be applied to real-world data sets. In simulation, both tests have high power relative to existing interpretability methods. When applied to state-of-the-art vision and language models, the framework selects features that intuitively explain model predictions. The resulting explanations have the additional advantage that they are themselves easy to interpret.