Exploring the Maze of Multilingual Modeling
This study provides insights for improving multilingual models, but it is incremental as it focuses on evaluating existing models without introducing new methods.
The paper evaluated three multilingual language models (mBERT, XLM-R, GPT-3) across languages to understand factors like resource availability and linguistic features affecting performance in text classification and generation tasks, finding that language-specific pretraining data is crucial but other factors also matter.
Multilingual language models have gained significant attention in recent years, enabling the development of applications that meet diverse linguistic contexts. In this paper, we present a comprehensive evaluation of three popular multilingual language models: mBERT, XLM-R, and GPT-3. We assess their performance across a diverse set of languages, with a focus on understanding the impact of resource availability (general and model-specific), language family, script type, and word order on model performance, under two distinct tasks - text classification and text generation. Our findings reveal that while the amount of language-specific pretraining data plays a crucial role in model performance, we also identify other factors such as general resource availability, language family, and script type, as important features. We hope that our study contributes to a deeper understanding of multilingual language models to enhance their performance across languages and linguistic contexts.