Toward LLMs Beyond English-Centric Development
For developers and researchers working on multilingual LLMs, the paper challenges the prevailing assumption that continual pre-training is an efficient adaptation strategy, highlighting the need for language-specific investments.
The paper shows that LLMs are heavily biased toward English and that continual pre-training for language adaptation is not cost-effective compared to training from scratch, even for cultural understanding. This implies that dedicated per-language investment may be necessary for future LLM development.
Through an analysis of sequences generated by open-weight large language models (LLMs), we demonstrate that LLMs are heavily biased toward English. While continual pre-training is commonly used to adapt LLMs to a target language, we show that it does not offer a cost advantage over training from scratch, even for improving cultural understanding in the target language. These findings suggest that dedicated per-language investment may become increasingly important for future LLM development, rather than relying primarily on the expansion of English-centric resources.