Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
This addresses the need for parameter-efficient multilingual models for resource-constrained scenarios, representing a strong incremental improvement over existing methods.
The paper tackles the problem of building efficient multilingual language models by moving beyond English-centric bitexts and using a novel sampling strategy, resulting in state-of-the-art performance across cross-lingual tasks with models that are 5-6x smaller than competitors while achieving 99.3% GLUE and 98.5% SQuAD 2.0 performance compared to English-only models.
In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing state-of-the-art models but are also more parameter efficient, thereby promoting better adoption in resource-constrained scenarios and practical applications. We show that going beyond English-centric bitexts, coupled with a novel sampling strategy aimed at reducing under-utilization of training data, substantially boosts performance across model sizes for both Electra and MLM pre-training objectives. We introduce XY-LENT: X-Y bitext enhanced Language ENcodings using Transformers which not only achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands, is also competitive across bands. Our XY-LENT XL variant outperforms XLM-RXXL and exhibits competitive performance with mT5 XXL while being 5x and 6x smaller respectively. We then show that our proposed method helps ameliorate the curse of multilinguality, with the XY-LENT XL achieving 99.3% GLUE performance and 98.5% SQuAD 2.0 performance compared to a SoTA English only model in the same size band. We then analyze our models performance on extremely low resource languages and posit that scaling alone may not be sufficient for improving the performance in this scenario