Language Fusion for Parameter-Efficient Cross-lingual Transfer
This addresses the issue of computational complexity in cross-lingual transfer for NLP practitioners, though it is incremental as it builds on existing adapter methods.
The paper tackles the problem of poor cross-lingual transfer performance due to undertrained representation spaces for non-English languages, proposing FLARE, a method that improves downstream task performance while maintaining parameter efficiency, achieving gains of 4.9% for Llama 3.1 and 2.2% for Gemma~2 on question-answering tasks.
Limited availability of multilingual text corpora for training language models often leads to poor performance on downstream tasks due to undertrained representation spaces for languages other than English. This 'under-representation' has motivated recent cross-lingual transfer methods to leverage the English representation space by e.g. mixing English and 'non-English' tokens at the input level or extending model parameters to accommodate new languages. However, these approaches often come at the cost of increased computational complexity. We propose Fusion forLanguage Representations (FLARE) in adapters, a novel method that enhances representation quality and downstream performance for languages other than English while maintaining parameter efficiency. FLARE integrates source and target language representations within low-rank (LoRA) adapters using lightweight linear transformations, maintaining parameter efficiency while improving transfer performance. A series of experiments across representative cross-lingual natural language understanding tasks, including natural language inference, question-answering and sentiment analysis, demonstrate FLARE's effectiveness. FLARE achieves performance improvements of 4.9% for Llama 3.1 and 2.2% for Gemma~2 compared to standard LoRA fine-tuning on question-answering tasks, as measured by the exact match metric.