Disentangling meaning from language in LLM-based machine translation
This work addresses the problem of understanding and controlling translation functions in LLMs for researchers in mechanistic interpretability and machine translation, though it is incremental as it builds on prior word-level analyses.
The study decomposed machine translation into language generation and meaning preservation, finding that distinct attention heads specialize in each subtask, and modifying just 1% of these heads enabled instruction-free translation performance comparable to instruction-based prompting.
Mechanistic Interpretability (MI) seeks to explain how neural networks implement their capabilities, but the scale of Large Language Models (LLMs) has limited prior MI work in Machine Translation (MT) to word-level analyses. We study sentence-level MT from a mechanistic perspective by analyzing attention heads to understand how LLMs internally encode and distribute translation functions. We decompose MT into two subtasks: producing text in the target language (i.e. target language identification) and preserving the input sentence's meaning (i.e. sentence equivalence). Across three families of open-source models and 20 translation directions, we find that distinct, sparse sets of attention heads specialize in each subtask. Based on this insight, we construct subtask-specific steering vectors and show that modifying just 1% of the relevant heads enables instruction-free MT performance comparable to instruction-based prompting, while ablating these heads selectively disrupts their corresponding translation functions.