LLM Parameters for Math Across Languages: Shared or Separate?
For researchers studying multilingual LLMs, this work clarifies the mechanistic basis of cross-lingual variation in mathematical reasoning, revealing a nuanced interplay between shared and language-specific parameters.
The paper investigates whether mathematical reasoning in multilingual LLMs relies on shared or language-specific parameters, finding partial cross-lingual overlap concentrated in intermediate layers, with English having the largest set of math-relevant parameters and lower-resource languages smaller sets.
Large language models (LLMs) exhibit substantial cross-lingual variation in mathematical reasoning performance, but it remains unclear whether these differences reflect language-specific parameters or a shared mechanism that manifests differently by language. We present a cross-lingual mechanistic analysis of mathematical reasoning in LLMs, enabling us to localize and compare model parameters that support mathematical reasoning across languages. We find that the extracted math-associated parameters exhibit partial cross-lingual overlap, with the strongest overlap concentrated in intermediate model layers. We further observe that English consistently produces the largest set of math-relevant parameters, whereas lower-resource languages reveal smaller sets of relevant parameters. These results suggest that math-related behavior in multilingual LLMs is neither fully language-invariant nor fully language-specific, but instead exhibits partial cross-lingual parameter overlap with systematic language-dependent differences.