Exploring Translation Mechanism of Large Language Models
This work addresses the problem of understanding and optimizing translation mechanisms in LLMs for researchers and practitioners, though it is incremental as it builds on existing analysis methods.
The study investigated the translation mechanisms of large language models, finding that translation is primarily driven by a sparse subset of specialized attention heads (less than 5%) and MLPs that process features into English-centric representations, with targeted fine-tuning of only 64 heads achieving translation improvements comparable to full-parameter tuning.
Large language models (LLMs) have succeeded remarkably in multilingual translation tasks. However, the inherent translation mechanisms of LLMs remain poorly understood, largely due to sophisticated architectures and vast parameter scales. In response to this issue, this study explores the translation mechanism of LLM from the perspective of computational components (e.g., attention heads and MLPs). Path patching is utilized to explore causal relationships between components, detecting those crucial for translation tasks and subsequently analyzing their behavioral patterns in human-interpretable terms. Comprehensive analysis reveals that translation is predominantly facilitated by a sparse subset of specialized attention heads (less than 5\%), which extract source language, indicator, and positional features. MLPs subsequently integrate and process these features by transiting towards English-centric latent representations. Notably, building on the above findings, targeted fine-tuning of only 64 heads achieves translation improvement comparable to full-parameter tuning while preserving general capabilities.