Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages
This addresses the problem of language inequity in AI for speakers of low-resource languages, offering a scalable and data-efficient solution, though it is incremental over existing methods like LangBridge.
The paper tackles the problem of large language models (LLMs) performing poorly on low-resource languages due to English-centric training, by proposing a novel architecture that fuses all intermediate layers of multilingual encoders to enrich linguistic information, resulting in improved accuracy from 71.66% to 75.86% on Sinhala classification and from 70.36% to 71.50% on average XNLI accuracy.
Large Language Models (LLMs) excel in English, but their performance degrades significantly on low-resource languages (LRLs) due to English-centric training. While methods like LangBridge align LLMs with multilingual encoders such as the Massively Multilingual Text-to-Text Transfer Transformer (mT5), they typically use only the final encoder layer. We propose a novel architecture that fuses all intermediate layers, enriching the linguistic information passed to the LLM. Our approach features two strategies: (1) a Global Softmax weighting for overall layer importance, and (2) a Transformer Softmax model that learns token-specific weights. The fused representations are mapped into the LLM's embedding space, enabling it to process multilingual inputs. The model is trained only on English data, without using any parallel or multilingual data. Evaluated on XNLI, IndicXNLI, Sinhala News Classification, and Amazon Reviews, our Transformer Softmax model significantly outperforms the LangBridge baseline. We observe strong performance gains in LRLs, improving Sinhala classification accuracy from 71.66% to 75.86% and achieving clear improvements across Indic languages such as Tamil, Bengali, and Malayalam. These specific gains contribute to an overall boost in average XNLI accuracy from 70.36% to 71.50%. This approach offers a scalable, data-efficient path toward more capable and equitable multilingual LLMs.