CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval
This addresses the underexplored area of code retrieval for developers and AI systems, offering a generalist solution that is incremental in advancing retrieval performance.
The paper tackles the problem of code retrieval across multiple programming languages and tasks by introducing CodeXEmbed, a family of large-scale embedding models, with the 7B model achieving over 20% improvement over the previous state-of-the-art on the CoIR benchmark.
Despite the success of text retrieval in many NLP tasks, code retrieval remains a largely underexplored area. Most text retrieval systems are tailored for natural language queries, often neglecting the specific challenges of retrieving code. This gap leaves existing models unable to effectively capture the diversity of programming languages and tasks across different domains, highlighting the need for more focused research in code retrieval. To address this, we introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters. Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework, enhancing model generalizability and retrieval performance. Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark. In addition to excelling in code retrieval, our models demonstrate competitive performance on the widely adopted BeIR text retrieval benchmark, offering versatility across domains. Experimental results demonstrate that improving retrieval performance significantly enhances end-to-end Retrieval-Augmented Generation (RAG) performance for code-related tasks.