The birth of Romanian BERT
This addresses the lack of high-performance language models for Romanian, an under-resourced language, by providing a dedicated model.
The paper introduces Romanian BERT, the first transformer-based language model pretrained solely on Romanian text, and reports its performance on various Romanian datasets, though no specific numbers are provided in the abstract.
Large-scale pretrained language models have become ubiquitous in Natural Language Processing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus composition and cleaning, the model training process, as well as an extensive evaluation of the model on various Romanian datasets. We open source not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process.