GREEK-BERT: The Greeks visiting Sesame Street
This addresses the problem of limited NLP resources for modern Greek, enabling improved research and applications in this domain, though it is incremental as it adapts an existing method to a new language.
The paper tackled the lack of high-performance language models for modern Greek by developing GREEK-BERT, a monolingual BERT-based model, which achieved state-of-the-art results in tasks like part-of-speech tagging and named entity recognition, outperforming multilingual models by 5%-10% in some benchmarks.
Transformer-based language models, such as BERT and its variants, have achieved state-of-the-art performance in several downstream natural language processing (NLP) tasks on generic benchmark datasets (e.g., GLUE, SQUAD, RACE). However, these models have mostly been applied to the resource-rich English language. In this paper, we present GREEK-BERT, a monolingual BERT-based language model for modern Greek. We evaluate its performance in three NLP tasks, i.e., part-of-speech tagging, named entity recognition, and natural language inference, obtaining state-of-the-art performance. Interestingly, in two of the benchmarks GREEK-BERT outperforms two multilingual Transformer-based models (M-BERT, XLM-R), as well as shallower neural baselines operating on pre-trained word embeddings, by a large margin (5%-10%). Most importantly, we make both GREEK-BERT and our training code publicly available, along with code illustrating how GREEK-BERT can be fine-tuned for downstream NLP tasks. We expect these resources to boost NLP research and applications for modern Greek.