The Latin Substrate: How Language Models Represent and Mediate Script Choice
This research provides insights into the internal mechanisms of LLMs for script variation, which is significant for developers working on multilingual models and transliteration tasks, offering a deeper understanding of how these models organize and mediate different writing systems.
This paper investigates how large language models (LLMs) handle languages written in multiple scripts, specifically focusing on transliteration. The authors found that LLMs consistently romanize content in latent representations during transliteration and can be steered to flip output scripts with a simple linear direction, which generalizes asymmetrically to unseen writing systems. They also localized a small set of late-layer attention heads that causally mediate script choice, transferring across unrelated languages and writing systems.
Many languages are written in multiple scripts, requiring large language models (LLMs) to generate equivalent linguistic content in distinct orthographic forms. While prior work suggests that LLMs route information through shared latent representations, how they internally mediate script variation remains poorly understood. We study this question by first examining per-layer output distributions with the logit lens, which reveals consistent latent romanization during transliteration, and then through representational and mechanistic analyses of script generation. At the representational level, we show that scripts of the same language become increasingly separable across layers and that a simple linear steering direction can flip a model's output script while largely maintaining semantic content. The vector generalizes asymmetrically to writing systems unseen during construction, flipping non-Latin output to Latin reliably, but mapping Latin output into varied non-Latin scripts. At the mechanistic level, we localize a small set of late-layer attention heads that causally mediate script choice. These heads transfer across unrelated languages and writing systems, suggesting that script routing is implemented by language-agnostic components. Across both analyses, we observe a consistent directional asymmetry: non-Latin output is produced by a compact, identifiable gate, while Latin-script output emerges from diffuse contributions across the network. Collectively, our findings hint that LLMs organize script variation around shared latent representations while exhibiting a privileged substrate toward Latin script.