TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models
This addresses a key limitation in multilingual NLP for languages with diverse scripts, offering a practical solution to enhance crosslingual transfer, though it is incremental as it builds on existing models.
The paper tackles the script barrier in multilingual pretrained language models, where different scripts hinder crosslingual transfer, by proposing TransliCo, a contrastive learning framework that fine-tunes models using transliterations to unify representations; it shows improved alignment and outperforms the original model on zero-shot tasks, with consistent gains in Indic languages.
The world's more than 7000 languages are written in at least 293 scripts. Due to various reasons, many closely related languages use different scripts, which poses a difficulty for multilingual pretrained language models (mPLMs) in learning crosslingual knowledge through lexical overlap. As a consequence, mPLMs are faced with a script barrier: representations from different scripts are located in different subspaces, which can result in crosslingual transfer involving languages of different scripts performing suboptimally. To address this problem, we propose TransliCo, a framework that optimizes the Transliteration Contrastive Modeling (TCM) objective to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script (in our case Latin), which enhances uniformity in the representation space for different scripts. Using Glot500-m, an mPLM pretrained on over 500 languages, as our source model, we fine-tune it on a small portion (5%) of its training data, and refer to the resulting model as Furina. We show that Furina not only better aligns representations from distinct scripts but also outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks. Additionally, we achieve consistent improvement in a case study on the Indic group where the languages exhibit areal features but use different scripts. We make our code and models publicly available.