VN-MTEB: Vietnamese Massive Text Embedding Benchmark
This addresses the problem of evaluating embedding models for Vietnamese applications, such as recommendation and content control, but is incremental as it adapts an existing benchmark to a new language.
The authors tackled the lack of large-scale test datasets for evaluating AI models in Vietnamese by introducing VN-MTEB, a benchmark created by translating English samples, which includes 41 datasets across six tasks and shows that models with Rotary Positional Embedding outperform those with Absolute Positional Embedding.
Vietnam ranks among the top countries in terms of both internet traffic and online toxicity. As a result, implementing embedding models for recommendation and content control duties in applications is crucial. However, a lack of large-scale test datasets, both in volume and task diversity, makes it tricky for scientists to effectively evaluate AI models before deploying them in real-world, large-scale projects. To solve this important problem, we introduce a Vietnamese benchmark, VN-MTEB for embedding models, which we created by translating a large number of English samples from the Massive Text Embedding Benchmark using our new automated framework. We leverage the strengths of large language models (LLMs) and cutting-edge embedding models to conduct translation and filtering processes to retain high-quality samples, guaranteeing a natural flow of language and semantic fidelity while preserving named entity recognition (NER) and code snippets. Our comprehensive benchmark consists of 41 datasets from six tasks specifically designed for Vietnamese text embeddings. In our analysis, we find that bigger and more complex models using Rotary Positional Embedding outperform those using Absolute Positional Embedding in embedding tasks. Datasets are available at HuggingFace: https://huggingface.co/collections/GreenNode/vn-mteb-68871433f0f7573b8e1a6686