Is the end of Insight in Sight ?
This work questions the feasibility and necessity of explainable AI in scientific contexts, highlighting a potential conflict between statistical interpolation and traditional insight.
The paper examines the tension between deep learning's predictive accuracy and the scientific ideal of mechanistic insight, using a physics-informed neural network on a gas dynamics problem, finding that the trained network's weights show no trace of physical principles despite accurate predictions.
The rise of deep learning challenges the longstanding scientific ideal of insight - the human capacity to understand phenomena by uncovering underlying mechanisms. In many modern applications, accurate predictions no longer require interpretable models, prompting debate about whether explainability is a realistic or even meaningful goal. From our perspective in physics, we examine this tension through a concrete case study: a physics-informed neural network (PINN) trained on a rarefied gas dynamics problem governed by the Boltzmann equation. Despite the system's clear structure and well-understood governing laws, the trained network's weights resemble Gaussian-distributed random matrices, with no evident trace of the physical principles involved. This suggests that deep learning and traditional simulation may follow distinct cognitive paths to the same outcome - one grounded in mechanistic insight, the other in statistical interpolation. Our findings raise critical questions about the limits of explainable AI and whether interpretability can - or should-remain a universal standard in artificial reasoning.