On the Challenges and Opportunities of Learned Sparse Retrieval for Code
This addresses the need for efficient and accurate code retrieval in LLM-based software engineering systems, representing a novel application of learned sparse retrieval to code.
The paper tackles the problem of retrieving code from large codebases using learned sparse retrieval, which is challenging due to issues like subword fragmentation and semantic gaps, and introduces SPLADE-Code models that achieve state-of-the-art performance (75.4 on MTEB Code for under 1B parameters) and enable sub-millisecond retrieval on a 1M-passage collection.
Retrieval over large codebases is a key component of modern LLM-based software engineering systems. Existing approaches predominantly rely on dense embedding models, while learned sparse retrieval (LSR) remains largely unexplored for code. However, applying sparse retrieval to code is challenging due to subword fragmentation, semantic gaps between natural-language queries and code, diversity of programming languages and sub-tasks, and the length of code documents, which can harm sparsity and latency. We introduce SPLADE-Code, the first large-scale family of learned sparse retrieval models specialized for code retrieval (600M-8B parameters). Despite a lightweight one-stage training pipeline, SPLADE-Code achieves state-of-the-art performance among retrievers under 1B parameters (75.4 on MTEB Code) and competitive results at larger scales (79.0 with 8B). We show that learned expansion tokens are critical to bridge lexical and semantic matching, and provide a latency analysis showing that LSR enables sub-millisecond retrieval on a 1M-passage collection with little effectiveness loss.