Does Transliteration Help Multilingual Language Modeling?
This addresses script diversity challenges for MLLMs in multilingual NLP, particularly for Indic languages, but is incremental as it applies an existing transliteration method to a specific context.
The study tackled the problem of script diversity hindering Multilingual Language Models (MLLMs) by transliterating closely related Indic languages to a common script, finding that it benefits low-resource languages without harming high-resource ones and improves cross-lingual representation similarity.
Script diversity presents a challenge to Multilingual Language Models (MLLM) by reducing lexical overlap among closely related languages. Therefore, transliterating closely related languages that use different writing scripts to a common script may improve the downstream task performance of MLLMs. We empirically measure the effect of transliteration on MLLMs in this context. We specifically focus on the Indic languages, which have the highest script diversity in the world, and we evaluate our models on the IndicGLUE benchmark. We perform the Mann-Whitney U test to rigorously verify whether the effect of transliteration is significant or not. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages. We also measure the cross-lingual representation similarity of the models using centered kernel alignment on parallel sentences from the FLORES-101 dataset. We find that for parallel sentences across different languages, the transliteration-based model learns sentence representations that are more similar.