Interpreting "Interpretability" and Explaining "Explainability" in Machine Learning in Physics
Provides conceptual clarity for physicists and ML practitioners navigating model transparency in scientific applications.
This review defines interpretability (structural transparency) and explainability (mapping to domain knowledge) in ML for physics, discussing trade-offs and tools. It argues these are deliberate modeling choices, not inherent properties.
We review the concepts of interpretability and explainability as they apply to machine learning in physics. We define interpretability as concerning the structural transparency of a model (the ability to understand or approximate its inner workings) and explainability as concerning the scientific content of a model (the ability to map it onto domain knowledge). We discuss the trade-offs each entails (interpretability vs. expressivity; explainability vs. adaptability), the contexts in which each is needed, and the intrinsic and post-hoc tools available for achieving them. Throughout, we emphasize that machine-learned models are subject to the same scientific questions as classical models, differing only in scale, and that interpretability and explainability are best understood as deliberate modeling choices rather than inherent properties. We also emphasize the importance of task specification and intervention plans as a core aspect of model design.