Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic
This reveals a fundamental mismatch in learning dynamics that affects reasoning reliability and robustness for AI systems, though it is incremental as it builds on prior work on skill composition discrepancies.
The study found that transformers learning arithmetic tasks do not follow human-like sequential skill composition, often acquiring skills in reverse or parallel order, leading to errors under distribution shifts, with this 'shattered compositionality' persisting in modern LLMs despite scaling or reasoning techniques.
Large language models (LLMs) often exhibit unexpected errors or unintended behavior, even at scale. While recent work reveals the discrepancy between LLMs and humans in skill compositions, the learning dynamics of skill compositions and the underlying cause of non-human behavior remain elusive. In this study, we investigate the mechanism of learning dynamics by training transformers on synthetic arithmetic tasks. Through extensive ablations and fine-grained diagnostic metrics, we discover that transformers do not reliably build skill compositions according to human-like sequential rules. Instead, they often acquire skills in reverse order or in parallel, which leads to unexpected mixing errors especially under distribution shifts--a phenomenon we refer to as shattered compositionality. To explain these behaviors, we provide evidence that correlational matching to the training data, rather than causal or procedural composition, shapes learning dynamics. We further show that shattered compositionality persists in modern LLMs and is not mitigated by pure model scaling or scratchpad-based reasoning. Our results reveal a fundamental mismatch between a model's learning behavior and desired skill compositions, with implications for reasoning reliability, out-of-distribution robustness, and alignment.