EstBERT: A Pretrained Language-Specific BERT for Estonian
This work addresses the need for better NLP tools for Estonian speakers, but it is incremental as it applies an existing method to a new language.
The paper tackles the problem of improving NLP performance for Estonian by developing EstBERT, a language-specific BERT model, and finds that it outperforms multilingual BERT models on five out of six tasks.
This paper presents EstBERT, a large pretrained transformer-based language-specific BERT model for Estonian. Recent work has evaluated multilingual BERT models on Estonian tasks and found them to outperform the baselines. Still, based on existing studies on other languages, a language-specific BERT model is expected to improve over the multilingual ones. We first describe the EstBERT pretraining process and then present the results of the models based on finetuned EstBERT for multiple NLP tasks, including POS and morphological tagging, named entity recognition and text classification. The evaluation results show that the models based on EstBERT outperform multilingual BERT models on five tasks out of six, providing further evidence towards a view that training language-specific BERT models are still useful, even when multilingual models are available.