Why Architecture Choice Matters in Symbolic Regression
For researchers using gradient-based symbolic regression, this paper highlights that architecture choice (tree structure) is a critical factor that can drastically affect recovery success, challenging the assumption that more expressive architectures are always better.
This paper demonstrates that in gradient-based symbolic regression, the choice of tree structure (how variables enter the tree) critically determines which targets are recovered, with one structure achieving 100% recovery on a target while another scores 0%, and rankings reversing across different targets. The most expressive structure can fail on targets that a restricted alternative solves reliably, showing that optimization landscape, not expressiveness alone, determines success.
Symbolic regression discovers mathematical formulas from data. Some methods fix a tree of operators, assign learnable weights, and train by gradient descent. The tree's structure, which determines what operators and variables appear at each position, is chosen once and applied to every target. This paper tests whether that choice affects which targets are actually recovered. Three structures are compared, all sharing the same operator and target language but differing in how variables enter the tree; one is strictly more expressive. Across over 12,700 training runs, one structure recovers a target at 100% while another scores 0%, and the ranking reverses on a different target. Expressiveness guarantees that a solution exists in the search space, but not that gradient descent finds it: the most expressive structure fails on targets that a restricted alternative solves reliably. Switching the operator changes which targets succeed; reversing its gradient profile collapses recovery entirely. Balanced (non-chain) tree shapes are never recovered. These findings show that the optimization landscape, not expressiveness alone, determines what gradient-based symbolic regression recovers.