PL-MTEB: Polish Massive Text Embedding Benchmark
This addresses the need for standardized evaluation of text embeddings in Polish for the NLP community, but it is incremental as it adapts existing tasks and datasets.
The authors tackled the lack of a comprehensive benchmark for text embeddings in Polish by introducing PL-MTEB, which includes 28 diverse NLP tasks and a new dataset, and they evaluated 15 models, providing aggregated results for task types and the entire benchmark.
In this paper, we introduce the Polish Massive Text Embedding Benchmark (PL-MTEB), a comprehensive benchmark for text embeddings in Polish. The PL-MTEB consists of 28 diverse NLP tasks from 5 task types. We adapted the tasks based on previously used datasets by the Polish NLP community. In addition, we created a new PLSC (Polish Library of Science Corpus) dataset consisting of titles and abstracts of scientific publications in Polish, which was used as the basis for two novel clustering tasks. We evaluated 15 publicly available models for text embedding, including Polish and multilingual ones, and collected detailed results for individual tasks and aggregated results for each task type and the entire benchmark. PL-MTEB comes with open-source code at https://github.com/rafalposwiata/pl-mteb.