Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus
This work addresses performance gaps in low-resource languages like Hindi for users of multilingual LLMs, though it is incremental as it builds on existing methods.
The authors tackled the problem of suboptimal performance of multilingual LLMs in low-resource languages by adapting a model to Hindi using continued pre-training and synthetic corpora, resulting in state-of-the-art results on Hindi benchmarks while maintaining competitiveness in English tasks.
Multilingual LLMs support a variety of languages; however, their performance is suboptimal for low-resource languages. In this work, we emphasize the importance of continued pre-training of multilingual LLMs and the use of translation-based synthetic pre-training corpora for improving LLMs in low-resource languages. We conduct our study in the context of the low-resource Indic language Hindi. We introduce Nemotron-Mini-Hindi 4B, a bilingual SLM supporting both Hindi and English, based on Nemotron-Mini 4B. The model is trained using a mix of real and synthetic Hindi + English tokens, with continuous pre-training performed on 400B tokens. We demonstrate that both the base and instruct models achieve state-of-the-art results on Hindi benchmarks while remaining competitive on English tasks. Additionally, we observe that the continued pre-training approach enhances the model's overall factual accuracy. We perform an ablation study to highlight the impact of Hindi pre-training, showing significant improvements in Hindi chat capabilities and factual accuracy, which cannot be achieved through Hindi alignment alone.