The Solvability of Interpretability Evaluation Metrics
This work addresses a fundamental issue in interpretability research by revealing a definition-evaluation duality, potentially impacting how explanations are designed and evaluated across domains like text, image, and tabular data.
The paper identifies that interpretability evaluation metrics like comprehensiveness and sufficiency are solvable via optimization (e.g., beam search), and shows that a beam search explainer outperforms existing methods like LIME, raising questions about why explainers aren't directly optimized for these metrics.
Feature attribution methods are popular for explaining neural network predictions, and they are often evaluated on metrics such as comprehensiveness and sufficiency. In this paper, we highlight an intriguing property of these metrics: their solvability. Concretely, we can define the problem of optimizing an explanation for a metric, which can be solved by beam search. This observation leads to the obvious yet unaddressed question: why do we use explainers (e.g., LIME) not based on solving the target metric, if the metric value represents explanation quality? We present a series of investigations showing strong performance of this beam search explainer and discuss its broader implication: a definition-evaluation duality of interpretability concepts. We implement the explainer and release the Python solvex package for models of text, image and tabular domains.