Dense Sparse Retrieval: Using Sparse Language Models for Inference Efficient Dense Retrieval
This addresses cost and management issues in academic and industrial search applications by making dense retrieval more efficient, though it is incremental as it adapts existing sparse models to a known task.
The paper tackled the problem of expensive GPU usage in dense retrieval systems by using sparse language models to improve inference efficiency, achieving up to 4.3x faster speeds with minimal accuracy loss on datasets like MSMARCO, NQ, and TriviaQA.
Vector-based retrieval systems have become a common staple for academic and industrial search applications because they provide a simple and scalable way of extending the search to leverage contextual representations for documents and queries. As these vector-based systems rely on contextual language models, their usage commonly requires GPUs, which can be expensive and difficult to manage. Given recent advances in introducing sparsity into language models for improved inference efficiency, in this paper, we study how sparse language models can be used for dense retrieval to improve inference efficiency. Using the popular retrieval library Tevatron and the MSMARCO, NQ, and TriviaQA datasets, we find that sparse language models can be used as direct replacements with little to no drop in accuracy and up to 4.3x improved inference speeds