Explaining the Explainer: Understanding the Inner Workings of Transformer-based Symbolic Regression Models
This work provides the first circuit-level characterization for symbolic regression transformers, establishing it as a promising domain for mechanistic interpretability research.
The authors tackled the problem of understanding how transformer-based symbolic regression models generate mathematical operators by introducing PATCHES, an evolutionary circuit discovery algorithm, which identified 28 circuits and showed that mean patching with performance-based evaluation reliably isolates functionally correct circuits.
Following their success across many domains, transformers have also proven effective for symbolic regression (SR); however, the internal mechanisms underlying their generation of mathematical operators remain largely unexplored. Although mechanistic interpretability has successfully identified circuits in language and vision models, it has not yet been applied to SR. In this article, we introduce PATCHES, an evolutionary circuit discovery algorithm that identifies compact and correct circuits for SR. Using PATCHES, we isolate 28 circuits, providing the first circuit-level characterisation of an SR transformer. We validate these findings through a robust causal evaluation framework based on key notions such as faithfulness, completeness, and minimality. Our analysis shows that mean patching with performance-based evaluation most reliably isolates functionally correct circuits. In contrast, we demonstrate that direct logit attribution and probing classifiers primarily capture correlational features rather than causal ones, limiting their utility for circuit discovery. Overall, these results establish SR as a high-potential application domain for mechanistic interpretability and propose a principled methodology for circuit discovery.