ParsBERT: Transformer-based Model for Persian Language Understanding
This addresses the problem of limited NLP resources for Persian language users, though it is incremental as it adapts an existing method to a new language.
The paper tackles the lack of monolingual pre-trained language models for Persian by proposing ParsBERT, a Transformer-based model, which achieves state-of-the-art performance in tasks like Sentiment Analysis, Text Classification, and Named Entity Recognition, outperforming multilingual BERT and other prior works.
The surge of pre-trained language models has begun a new era in the field of Natural Language Processing (NLP) by allowing us to build powerful language models. Among these models, Transformer-based models such as BERT have become increasingly popular due to their state-of-the-art performance. However, these models are usually focused on English, leaving other languages to multilingual models with limited resources. This paper proposes a monolingual BERT for the Persian language (ParsBERT), which shows its state-of-the-art performance compared to other architectures and multilingual models. Also, since the amount of data available for NLP tasks in Persian is very restricted, a massive dataset for different NLP tasks as well as pre-training the model is composed. ParsBERT obtains higher scores in all datasets, including existing ones as well as composed ones and improves the state-of-the-art performance by outperforming both multilingual BERT and other prior works in Sentiment Analysis, Text Classification and Named Entity Recognition tasks.