Bilingual Language Modeling, A transfer learning technique for Roman Urdu
This addresses the problem of limited NLP capabilities for low-resource languages like Roman Urdu, though it is incremental as it builds on existing transfer learning and code-switching concepts.
The paper tackles the challenge of applying pretrained language models to low-resource languages like Roman Urdu by proposing Bilingual Language Modeling, a transfer learning technique that leverages code-switching from a high-resource language, resulting in a bilingual model achieving 23% accuracy in Masked Language Modeling compared to 2% for monolingual and 11% for multilingual models.
Pretrained language models are now of widespread use in Natural Language Processing. Despite their success, applying them to Low Resource languages is still a huge challenge. Although Multilingual models hold great promise, applying them to specific low-resource languages e.g. Roman Urdu can be excessive. In this paper, we show how the code-switching property of languages may be used to perform cross-lingual transfer learning from a corresponding high resource language. We also show how this transfer learning technique termed Bilingual Language Modeling can be used to produce better performing models for Roman Urdu. To enable training and experimentation, we also present a collection of novel corpora for Roman Urdu extracted from various sources and social networking sites, e.g. Twitter. We train Monolingual, Multilingual, and Bilingual models of Roman Urdu - the proposed bilingual model achieves 23% accuracy compared to the 2% and 11% of the monolingual and multilingual models respectively in the Masked Language Modeling (MLM) task.