Neural Code Search Evaluation Dataset
This work provides a standardized evaluation dataset for researchers in code search, but it is incremental as it builds on existing models without introducing new methods.
The authors tackled the lack of a common benchmark for evaluating code search models by creating a dataset of natural language query and code snippet pairs, and they provided results from two existing models to establish baseline performance.
There has been an increase of interest in code search using natural language. Assessing the performance of such code search models can be difficult without a readily available evaluation suite. In this paper, we present an evaluation dataset consisting of natural language query and code snippet pairs, with the hope that future work in this area can use this dataset as a common benchmark. We also provide the results of two code search models ([1] and [6]) from recent work. The evaluation dataset is available at https://github.com/facebookresearch/Neural-Code-Search-Evaluation-Dataset