In-Context Algebra
This work addresses the challenge of symbolic reasoning in AI for researchers, showing that transformers can learn structured mechanisms in variable-based tasks, though it is incremental in building on prior studies of in-context learning.
The paper tackled the problem of understanding how transformers learn to perform algebraic reasoning when token meanings vary in-context, and found that they achieve near-perfect accuracy and generalize to unseen groups by developing specific mechanisms like commutative copying and closure-based cancellation.
We investigate the mechanisms that arise when transformers are trained to solve arithmetic on sequences where tokens are variables whose meaning is determined only through their interactions in-context. While prior work has studied transformers in settings where the answer relies on fixed parametric or geometric information encoded in token embeddings, we devise a new in-context reasoning task where the assignment of tokens to specific algebraic elements varies from one sequence to another. Despite this challenging setup, transformers achieve near-perfect accuracy on the task and even generalize to unseen groups. We develop targeted data distributions to create causal tests of a set of hypothesized mechanisms, and we isolate three mechanisms models consistently learn: commutative copying where a dedicated head copies answers, identity element recognition that distinguishes identity-containing facts, and closure-based cancellation that tracks group membership to constrain valid answers. Our findings show that the kinds of reasoning strategies learned by transformers are dependent on the task structure and that models can develop symbolic reasoning mechanisms when trained to reason in-context about variables whose meanings are not fixed.