Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages
This work addresses computational inefficiencies in adapting multilingual models for specific languages, offering incremental improvements over existing methods.
The study tackled the problem of adapting multilingual language models to new languages by replacing cross-lingual vocabularies with compact, language-specific ones, demonstrating that simple embedding re-initialization techniques rival more complex methods and improve efficiency for low-resource languages.
Pre-trained multilingual language models underpin a large portion of modern NLP tools outside of English. A strong baseline for specializing these models for specific languages is Language-Adaptive Pre-Training (LAPT). However, retaining a large cross-lingual vocabulary and embedding matrix comes at considerable excess computational cost during adaptation. In this study, we propose several simple techniques to replace a cross-lingual vocabulary with a compact, language-specific one. Namely, we address strategies for re-initializing the token embedding matrix after vocabulary specialization. We then provide a systematic experimental comparison of our techniques, in addition to the recently-proposed Focus method. We demonstrate that: 1) Embedding-replacement techniques in the monolingual transfer literature are inadequate for adapting multilingual models. 2) Replacing cross-lingual vocabularies with smaller specialized ones provides an efficient method to improve performance in low-resource languages. 3) Simple embedding re-initialization techniques based on script-wise sub-distributions rival techniques such as Focus, which rely on similarity scores obtained from an auxiliary model.