Assessing the Role of Data Quality in Training Bilingual Language Models
This addresses the problem of balancing performance across languages in multilingual NLP, offering a practical solution for researchers and practitioners, though it is incremental as it builds on existing data quality insights.
The study tackled performance inconsistencies in bilingual language models by identifying unequal data quality as a key driver, and proposed a data filtering strategy that improved monolingual performance by 2-4% and reduced bilingual gaps to 1% for French, German, and Chinese.
Bilingual and multilingual language models offer a promising path toward scaling NLP systems across diverse languages and users. However, their performance often varies wildly between languages as prior works show that adding more languages can degrade performance for some languages (such as English), while improving others (typically more data constrained languages). In this work, we investigate causes of these inconsistencies by comparing bilingual and monolingual language models. Our analysis reveals that unequal data quality, not just data quantity, is a major driver of performance degradation in bilingual settings. We propose a simple yet effective data filtering strategy to select higher-quality bilingual training data with only high quality English data. Applied to French, German, and Chinese, our approach improves monolingual performance by 2-4% and reduces bilingual model performance gaps to 1%. These results highlight the overlooked importance of data quality in multilingual pretraining and offer a practical recipe for balancing performance.