GreekBART: The First Pretrained Greek Sequence-to-Sequence Model
This provides a specialized tool for Greek NLP tasks, addressing a language gap, but it is incremental as it adapts an existing architecture to a new language.
The authors tackled the lack of pretrained sequence-to-sequence models for Greek by introducing GreekBART, a BART-based model pretrained on a large Greek corpus, which outperformed baselines like Greek-BERT and XLM-R on discriminative tasks and showed strong results on new Greek summarization tasks.
The era of transfer learning has revolutionized the fields of Computer Vision and Natural Language Processing, bringing powerful pretrained models with exceptional performance across a variety of tasks. Specifically, Natural Language Processing tasks have been dominated by transformer-based language models. In Natural Language Inference and Natural Language Generation tasks, the BERT model and its variants, as well as the GPT model and its successors, demonstrated exemplary performance. However, the majority of these models are pretrained and assessed primarily for the English language or on a multilingual corpus. In this paper, we introduce GreekBART, the first Seq2Seq model based on BART-base architecture and pretrained on a large-scale Greek corpus. We evaluate and compare GreekBART against BART-random, Greek-BERT, and XLM-R on a variety of discriminative tasks. In addition, we examine its performance on two NLG tasks from GreekSUM, a newly introduced summarization dataset for the Greek language. The model, the code, and the new summarization dataset will be publicly available.