SINA-BERT: A pre-trained Language Model for Analysis of Medical Texts in Persian
This addresses the problem of limited resources for Persian medical text analysis, though it is incremental as it adapts an existing method to a new domain and language.
The authors tackled the lack of a high-quality Persian language model for medical texts by releasing SINA-BERT, a BERT-based model pre-trained on a large-scale corpus of formal and informal medical content, which outperforms existing Persian BERT models on tasks like medical question categorization, sentiment analysis, and question retrieval.
We have released Sina-BERT, a language model pre-trained on BERT (Devlin et al., 2018) to address the lack of a high-quality Persian language model in the medical domain. SINA-BERT utilizes pre-training on a large-scale corpus of medical contents including formal and informal texts collected from a variety of online resources in order to improve the performance on health-care related tasks. We employ SINA-BERT to complete following representative tasks: categorization of medical questions, medical sentiment analysis, and medical question retrieval. For each task, we have developed Persian annotated data sets for training and evaluation and learnt a representation for the data of each task especially complex and long medical questions. With the same architecture being used across tasks, SINA-BERT outperforms BERT-based models that were previously made available in the Persian language.