OASIS: Order-Augmented Strategy for Improved Code Search
This work addresses the challenge of improving code search accuracy for developers using LLMs, representing an incremental advancement by focusing on training with negative pair differences.
The paper tackles the problem of capturing deeper semantic nuances in code embeddings for code search by proposing an order-augmented strategy (OASIS) that uses order-based similarity labels to train models on subtle differences among negative pairs, resulting in significant outperformance over previous state-of-the-art models in benchmark evaluations.
Code embeddings capture the semantic representations of code and are crucial for various code-related large language model (LLM) applications, such as code search. Previous training primarily relies on optimizing the InfoNCE loss by comparing positive natural language (NL)-code pairs with in-batch negatives. However, due to the sparse nature of code contexts, training solely by comparing the major differences between positive and negative pairs may fail to capture deeper semantic nuances. To address this issue, we propose a novel order-augmented strategy for improved code search (OASIS). It leverages order-based similarity labels to train models to capture subtle differences in similarity among negative pairs. Extensive benchmark evaluations demonstrate that our OASIS model significantly outperforms previous state-of-the-art models focusing solely on major positive-negative differences. It underscores the value of exploiting subtle differences among negative pairs with order labels for effective code embedding training.