Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models
For scientists and practitioners using ML to infer causal or structural knowledge, the paper highlights a fundamental limitation of post-hoc explanations that is often overlooked.
The paper argues that post-hoc explanations of scientific ML models cannot, on their own, support claims about the structure of the phenomenon, even if the model is reliable and the explanation faithful. This is because neither reliability nor faithfulness checks whether the model works as the phenomenon works.
Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. Reliability checks that the model's predictions match the phenomenon's outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works, which is what a claim about structure requires. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.