NV-Retriever: Improving text embedding models with effective hard-negative mining
This work addresses a bottleneck in training text embedding models for information retrieval applications like semantic search and RAG, offering incremental improvements in hard-negative mining.
The paper tackles the challenge of selecting high-quality hard-negative passages for fine-tuning text embedding models in contrastive learning, introducing positive-aware mining methods that improve training efficiency and retrieval accuracy, resulting in a model scoring 60.9 on the MTEB Retrieval benchmark and achieving first place upon publication.
Text embedding models have been popular for information retrieval applications such as semantic search and Question-Answering systems based on Retrieval-Augmented Generation (RAG). Those models are typically Transformer models that are fine-tuned with contrastive learning objectives. One of the challenging aspects of fine-tuning embedding models is the selection of high quality hard-negative passages for contrastive learning. In this paper we introduce a family of positive-aware mining methods that use the positive relevance score as an anchor for effective false negative removal, leading to faster training and more accurate retrieval models. We provide an ablation study on hard-negative mining methods over their configurations, exploring different teacher and base models. We further demonstrate the efficacy of our proposed mining methods at scale with the NV-Retriever-v1 model, which scores 60.9 on MTEB Retrieval (BEIR) benchmark and placed 1st when it was published to the MTEB Retrieval on July, 2024.