Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
For mechanistic interpretability researchers, this work reveals that language models can repurpose generic arithmetic circuits for reasoning about cyclic concepts, challenging assumptions about direct structure-computation correspondence.
Llama-3.1-8B reasons about cyclic concepts (e.g., months) by first performing base-10 addition on the inputs, then mapping the sum back to the cyclic space, rather than directly computing modular addition. The model re-uses a generic addition mechanism with task-agnostic Fourier features (periods 2, 5, 10) and a sparse set of 28 MLP neurons (0.2% of layer 18) across tasks.
Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even though Llama-3.1-8B's representations for these concepts are circularly structured, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using base-10 addition (six + August=14). Then, it maps this sum back to cyclic concept space (14->February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums--in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12 for months). Furthermore, we identify a sparse set of 28 MLP neurons re-used across all tasks (approximately 0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters, each computing the sum for a Fourier feature with a different period. Our work highlights how an interplay between causal abstraction and feature geometry can deepen our mechanistic understanding of LMs.