LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation
This addresses a gap in real-world software development for developers using fast-evolving libraries, though it is incremental as it builds on existing code completion research.
The authors tackled the problem of code completion models failing to account for rapidly-evolving public libraries by introducing LibEvolutionEval, a benchmark for version-specific code generation, and found that library evolution significantly impacts model performance, with mitigation methods like retrieved documentation and prompting showing promise.
Recent advancements in code completion models have primarily focused on local file contexts. However, these studies do not fully capture the complexity of real-world software development, which often requires the use of rapidly-evolving public libraries. To fill the gap, we introduce LibEvolutionEval, a detailed study requiring an understanding of library evolution to perform in-line code completion accurately. LibEvolutionEval provides a version-specific code-completion task comprised of eight libraries (torch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas) as they evolve over the year along with a detailed analysis of the evolution of two popular and well-maintained public libraries: PyTorch and Matplotlib. We evaluate popular public models and find that public library evolution significantly influences model performance. We explored mitigation methods by studying how retrieved version-specific library documentation and prompting can improve the model's capability in handling these fast-evolving packages, paving a promising future path in better handling fast-evolving libraries.