Slovene SuperGLUE Benchmark: Translation and Evaluation
This work provides a benchmark for Slovene natural language processing, but it is incremental as it adapts an existing benchmark to a new language.
The authors tackled the problem of creating a Slovene version of the SuperGLUE benchmark by translating it and evaluating performance across monolingual, cross-lingual, and multilingual modes, finding that the monolingual Slovene SloBERTa model outperforms multilingual models but still lags behind English models.
We present a Slovene combined machine-human translated SuperGLUE benchmark. We describe the translation process and problems arising due to differences in morphology and grammar. We evaluate the translated datasets in several modes: monolingual, cross-lingual, and multilingual, taking into account differences between machine and human translated training sets. The results show that the monolingual Slovene SloBERTa model is superior to massively multilingual and trilingual BERT models, but these also show a good cross-lingual performance on certain tasks. The performance of Slovene models still lags behind the best English models.