Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models
This addresses the need for efficient and interpretable multilingual control in AI systems, offering a domain-specific improvement over existing methods.
The paper tackled the problem of controlling multilingual output in large language models by identifying sparse dimensions that govern cross-lingual transitions, resulting in a method that can switch output languages while preserving semantics and outperforms prior neuron-based approaches with lower cost, using as few as 50 sentences.
Large language models exhibit strong multilingual capabilities despite limited exposure to non-English data. Prior studies show that English-centric large language models map multilingual content into English-aligned representations at intermediate layers and then project them back into target-language token spaces in the final layer. From this observation, we hypothesize that this cross-lingual transition is governed by a small and sparse set of dimensions, which occur at consistent indices across the intermediate to final layers. Building on this insight, we introduce a simple, training-free method to identify and manipulate these dimensions, requiring only as few as 50 sentences of either parallel or monolingual data. Experiments on a multilingual generation control task reveal the interpretability of these dimensions, demonstrating that the interventions in these dimensions can switch the output language while preserving semantic content, and that it surpasses the performance of prior neuron-based approaches at a substantially lower cost.