ReCoVeR the Target Language: Language Steering without Sacrificing Task Performance
This addresses the problem of unwanted language switching in LLMs for users requiring consistent language output, representing a novel improvement over prior methods that sacrificed task performance.
The paper tackles language confusion in multilingual Large Language Models (LLMs) by proposing ReCoVeR, a lightweight method using language-specific steering vectors, which effectively reduces confusion across 18 languages in benchmarks while maintaining task performance.
As they become increasingly multilingual, Large Language Models (LLMs) exhibit more language confusion, i.e., they tend to generate answers in a language different from the language of the prompt or the answer language explicitly requested by the user. In this work, we propose ReCoVeR (REducing language COnfusion in VEctor Representations), a novel lightweight approach for reducing language confusion based on language-specific steering vectors. We first isolate language vectors with the help of multi-parallel corpus and then effectively leverage those vectors for effective LLM steering via fixed (i.e., unsupervised) as well as trainable steering functions. Our extensive evaluation, encompassing three benchmarks and 18 languages, shows that ReCoVeR effectively mitigates language confusion in both monolingual and cross-lingual setups while at the same time -- and in contrast to prior language steering methods -- retaining task performance. Our data code is available at https://github.com/hSterz/recover.