LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference
This addresses efficiency issues for deploying LLM-based retrieval systems in real-time applications, though it is incremental as it builds on existing LLM retrieval methods.
The paper tackles the problem of slow query inference in LLM-based text retrieval by proposing LightRetriever, which uses lightweight query encoders to achieve over 1000x speedup in query encoding and over 10x increase in end-to-end retrieval throughput while maintaining 95% retrieval performance on benchmarks.
Large Language Models (LLMs)-based text retrieval retrieves documents relevant to search queries based on vector similarities. Documents are pre-encoded offline, while queries arrive in real-time, necessitating an efficient online query encoder. Although LLMs significantly enhance retrieval capabilities, serving deeply parameterized LLMs slows down query inference throughput and increases demands for online deployment resources. In this paper, we propose LightRetriever, a novel LLM-based retriever with extremely lightweight query encoders. Our method retains a full-sized LLM for document encoding, but reduces the workload of query encoding to no more than an embedding lookup. Compared to serving a full LLM on an A800 GPU, our method achieves over 1000x speedup in query encoding and over 10x increase in end-to-end retrieval throughput. Extensive experiments on large-scale retrieval benchmarks show that LightRetriever generalizes well across diverse tasks, maintaining an average of 95% retrieval performance.