The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding
This addresses the evaluation gap for Scandinavian languages in text embedding, though it is incremental as it extends existing benchmarking efforts like MTEB to a specific domain.
The authors tackled the lack of multilingual text embedding benchmarks by introducing the Scandinavian Embedding Benchmark (SEB), a comprehensive framework for evaluating Scandinavian languages across 24 tasks, and found significant performance disparities among over 26 models, including public and commercial solutions.
The evaluation of English text embeddings has transitioned from evaluating a handful of datasets to broad coverage across many tasks through benchmarks such as MTEB. However, this is not the case for multilingual text embeddings due to a lack of available benchmarks. To address this problem, we introduce the Scandinavian Embedding Benchmark (SEB). SEB is a comprehensive framework that enables text embedding evaluation for Scandinavian languages across 24 tasks, 10 subtasks, and 4 task categories. Building on SEB, we evaluate more than 26 models, uncovering significant performance disparities between public and commercial solutions not previously captured by MTEB. We open-source SEB and integrate it with MTEB, thus bridging the text embedding evaluation gap for Scandinavian languages.