Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention
This addresses language disparities in LLMs for multilingual NLP applications, offering a cost-effective alternative to resource-intensive methods like pretraining or fine-tuning.
The paper tackles performance gaps in large language models across different languages by proposing INCLINE, a framework that aligns internal representations during inference, resulting in significant improvements on nine benchmarks with five LLMs compared to strong baselines.
Large Language Models (LLMs) have shown remarkable capabilities in natural language processing but exhibit significant performance gaps among different languages. Most existing approaches to address these disparities rely on pretraining or fine-tuning, which are resource-intensive. To overcome these limitations without incurring significant costs, we propose Inference-Time Cross-Lingual Intervention (INCLINE), a novel framework that enhances LLM performance on low-performing (source) languages by aligning their internal representations with those of high-performing (target) languages during inference. INCLINE initially learns alignment matrices using parallel sentences from source and target languages through a Least-Squares optimization, and then applies these matrices during inference to transform the low-performing language representations toward the high-performing language space. Extensive experiments on nine benchmarks with five LLMs demonstrate that INCLINE significantly improves performance across diverse tasks and languages, compared to recent strong baselines. Our analysis demonstrates that INCLINE is highly cost-effective and applicable to a wide range of applications. In addition, we release the code to foster research along this line: https://github.com/weixuan-wang123/INCLINE.