The Impact of Model Scaling on Seen and Unseen Language Performance
This research addresses the need for understanding performance disparities in multilingual LLMs, which is crucial for developers and users handling diverse languages, though it is incremental in analyzing scaling effects.
The study investigated how model scaling affects multilingual large language models' performance in text classification and machine translation across 204 languages, finding that scale has little impact in zero-shot settings but leads to linear improvements in two-shot text classification, with resource levels being key predictors of effectiveness.
The rapid advancement of Large Language Models (LLMs), particularly those trained on multilingual corpora, has intensified the need for a deeper understanding of their performance across a diverse range of languages and model sizes. Our research addresses this critical need by studying the performance and scaling behavior of multilingual LLMs in text classification and machine translation tasks across 204 languages. We systematically examine both seen and unseen languages across three model families of varying sizes in zero-shot and few-shot settings. Our findings show significant differences in scaling behavior between zero-shot and two-shot scenarios, with striking disparities in performance between seen and unseen languages. Model scale has little effect on zero-shot performance, which remains mostly flat. However, in two-shot settings, larger models show clear linear improvements in multilingual text classification. For translation tasks, however, only the instruction-tuned model showed clear benefits from scaling. Our analysis also suggests that overall resource levels, not just the proportions of pretraining languages, are better predictors of model performance, shedding light on what drives multilingual LLM effectiveness.