FinEst BERT and CroSloEngual BERT: less is more in multilingual models
This work addresses the need for better NLP models for specific low-resource language groups, though it is incremental as it applies an existing method to new language combinations.
The authors tackled the problem of improving NLP performance for low-resource languages by training trilingual BERT models (FinEst BERT and CroSloEngual BERT) for Finnish, Estonian, Croatian, Slovenian, and English, which improved results on tasks like NER, POS-tagging, and dependency parsing compared to multilingual baselines.
Large pretrained masked language models have become state-of-the-art solutions for many NLP problems. The research has been mostly focused on English language, though. While massively multilingual models exist, studies have shown that monolingual models produce much better results. We train two trilingual BERT-like models, one for Finnish, Estonian, and English, the other for Croatian, Slovenian, and English. We evaluate their performance on several downstream tasks, NER, POS-tagging, and dependency parsing, using the multilingual BERT and XLM-R as baselines. The newly created FinEst BERT and CroSloEngual BERT improve the results on all tasks in most monolingual and cross-lingual situations