Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
This work addresses the challenge of making vision-language models effective across diverse languages, which is crucial for global accessibility, though it is incremental as it builds on existing multilingual training approaches with systematic analysis.
The paper tackled the problem of multilingual ability in large vision-language models, which often struggle with non-English inputs due to English-centric training, by systematically investigating training strategies across 13 tasks and 43 languages; it found that including up to 100 languages with 25-50% non-English data improves multilingual performance while maintaining English performance, and trained Centurio, a 100-language model achieving state-of-the-art results on 14 tasks and 56 languages.
Most Large Vision-Language Models (LVLMs) to date are trained predominantly on English data, which makes them struggle to understand non-English input and fail to generate output in the desired target language. Existing efforts mitigate these issues by adding multilingual training data, but do so in a largely ad-hoc manner, lacking insight into how different training mixes tip the scale for different groups of languages. In this work, we present a comprehensive investigation into the training strategies for massively multilingual LVLMs. First, we conduct a series of multi-stage experiments spanning 13 downstream vision-language tasks and 43 languages, systematically examining: (1) the number of training languages that can be included without degrading English performance and (2) optimal language distributions of pre-training as well as (3) instruction-tuning data. Further, we (4) investigate how to improve multilingual text-in-image understanding, and introduce a new benchmark for the task. Surprisingly, our analysis reveals that one can (i) include as many as 100 training languages simultaneously (ii) with as little as 25-50\% of non-English data, to greatly improve multilingual performance while retaining strong English performance. We further find that (iii) including non-English OCR data in pre-training and instruction-tuning is paramount for improving multilingual text-in-image understanding. Finally, we put all our findings together and train Centurio, a 100-language LVLM, offering state-of-the-art performance in an evaluation covering 14 tasks and 56 languages.