What is my math transformer doing? -- Three results on interpretability and generalization
This addresses interpretability and generalization concerns for transformers in mathematical tasks, providing insights into their reliability and debunking myths about hallucination and memorization, which is incremental but valuable for researchers in AI and mathematics.
The paper investigates failure cases and out-of-distribution behavior of transformers trained on matrix inversion and eigenvalue decomposition, showing that incorrect predictions retain key mathematical properties like correct eigenvalues and unit norm of eigenvectors, and that failures are predictable from problem or solution properties, with careful dataset choice accelerating training and enabling generalization beyond the training distribution.
This paper investigates the failure cases and out-of-distribution behavior of transformers trained on matrix inversion and eigenvalue decomposition. I show that incorrect model predictions still retain deep mathematical properties of the solution (e.g. correct eigenvalues, unit norm of eigenvectors), and that almost all model failures can be attributed to, and predicted from, properties of the problem or solution. This demonstrates that, when in doubt, math transformers do not hallucinate absurd solutions (as was sometimes proposed) but remain ``roughly right''. I also show that the careful choice of a training dataset can accelerate training, while allowing the model to generalize out of its training distribution, invalidating the idea that transformers ``merely interpolate'' from memorized examples.