Specializing Multilingual Language Models: An Empirical Study
This work addresses the problem of adapting NLP tools to low-resource languages, but it is incremental as it builds on existing adaptation methods without introducing a new paradigm.
The study investigated the performance, extensibility, and interaction of vocabulary augmentation and script transliteration for adapting multilingual language models to low-resource languages, finding these approaches viable in evaluations on part-of-speech tagging, dependency parsing, and named entity recognition across nine languages.
Pretrained multilingual language models have become a common tool in transferring NLP capabilities to low-resource languages, often with adaptations. In this work, we study the performance, extensibility, and interaction of two such adaptations: vocabulary augmentation and script transliteration. Our evaluations on part-of-speech tagging, universal dependency parsing, and named entity recognition in nine diverse low-resource languages uphold the viability of these approaches while raising new questions around how to optimally adapt multilingual models to low-resource settings.