Development of Pre-Trained Transformer-based Models for the Nepali Language
This addresses the lack of resources for Nepali speakers by providing improved models for understanding and generating text, though it is incremental as it applies existing methods to a new language.
The paper tackled the underrepresentation of the Nepali language in NLP by developing pre-trained transformer models (BERT, RoBERTa, GPT-2) using a new 27.5 GB corpus, resulting in a 2-point improvement to 95.60 on the Nep-gLUE benchmark and better text generation.
Transformer-based pre-trained language models have dominated the field of Natural Language Processing (NLP) for quite some time now. However, the Nepali language, spoken by approximately 32 million people worldwide, remains significantly underrepresented in this domain. This underrepresentation is primarily attributed to the scarcity of monolingual data corpora and limited available resources for the Nepali language. While existing efforts have predominantly concentrated on basic encoder-based models, there is a notable gap in the exploration of decoder-based architectures. To address this gap, we have collected 27.5 GB of Nepali text data, approximately 2.4x larger than any previously available Nepali language corpus. Leveraging this data, we pre-trained three different models i.e., BERT, RoBERTa, and GPT-2, exclusively for the Nepali Language. Furthermore, we performed instruction tuning and explored its potential for monolingual Nepali data, providing a foundation for future research. Our models outperformed the existing best model by 2 points on Nep-gLUE benchmark, scoring 95.60 and also outperformed existing models on text generation tasks, demonstrating improvements in both understanding and generating Nepali text.