Sequence to sequence pretraining for a less-resourced Slovenian language
This work addresses the problem of limited NLP resources for Slovenian speakers, but it is incremental as it applies an existing method to a new language.
The researchers tackled the lack of pretrained sequence-to-sequence models for Slovenian, a less-resourced language, by training two T5-type models and evaluating them on 11 tasks, finding they lag behind SloBERTa in classification but are useful for generative tasks.
Large pretrained language models have recently conquered the area of natural language processing. As an alternative to predominant masked language modelling introduced in BERT, the T5 model has introduced a more general training objective, namely sequence to sequence transformation, which includes masked language model but more naturally fits text generation tasks such as machine translation, summarization, question answering, text simplification, dialogue systems, etc. The monolingual variants of T5 models have been limited to well-resourced languages, while the massively multilingual T5 model supports 101 languages. In contrast, we trained two different sized T5-type sequence to sequence models for morphologically rich Slovene language with much less resources and analyzed their behavior on 11 tasks. Concerning classification tasks, the SloT5 models mostly lag behind the monolingual Slovene SloBERTa model but are useful for the generative tasks.