VIBE: Vector Index Benchmark for Embeddings
This provides an up-to-date benchmarking tool for researchers and practitioners in machine learning, addressing a critical need for evaluating ANN algorithms in applications like retrieval-augmented generation, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of outdated benchmarks for approximate nearest neighbor (ANN) search by introducing VIBE, an open-source benchmark that includes modern datasets and out-of-distribution scenarios, resulting in a comprehensive evaluation of 21 vector index implementations on 18 datasets.
Approximate nearest neighbor (ANN) search is a performance-critical component of many machine learning pipelines. Rigorous benchmarking is essential for evaluating the performance of vector indexes for ANN search. However, the datasets of the existing benchmarks are no longer representative of the current applications of ANN search. Hence, there is an urgent need for an up-to-date set of benchmarks. To this end, we introduce Vector Index Benchmark for Embeddings (VIBE), an open source project for benchmarking ANN algorithms. VIBE contains a pipeline for creating benchmark datasets using dense embedding models characteristic of modern applications, such as retrieval-augmented generation (RAG). To replicate real-world workloads, we also include out-of-distribution (OOD) datasets where the queries and the corpus are drawn from different distributions. We use VIBE to conduct a comprehensive evaluation of SOTA vector indexes, benchmarking 21 implementations on 12 in-distribution and 6 out-of-distribution datasets.