SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation
For researchers and practitioners working on Slovak NLP, this provides the first comprehensive embedding benchmark and efficient, locally-deployable models for semantic search and RAG.
The authors introduce SkMTEB, the first comprehensive MTEB-style benchmark for Slovak with 31 datasets across 7 task types, and develop efficient Slovak embedding models (e5-sk-small and e5-sk-large) via vocabulary trimming and fine-tuning of Multilingual E5, achieving competitive performance with proprietary APIs despite up to 62% size reduction.
We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types -- nearly 4$\times$ the depth of existing multilingual benchmark coverage for Slovak. Our evaluation of 31 embedding models reveals that large instruction-tuned multilingual models achieve the strongest performance, while existing Slovak-specific models trained for NLU tasks transfer poorly to embedding tasks. To address the need for efficient, locally-deployable Slovak embeddings, we develop \texttt{e5-sk-small} (45M parameters) and \texttt{e5-sk-large} (365M) by applying vocabulary trimming and fine-tuning to Multilingual E5 models. Despite size reductions of up to 62\%, our open-source models achieve competitive performance with proprietary APIs while remaining locally deployable for semantic search and retrieval-augmented generation (RAG). We release the benchmark, models, datasets, and code openly, hoping our approach offers a replicable path for other under-resourced languages.