LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations
This addresses the problem of library migration recommendations for software developers, though it appears incremental as it applies existing RAG and LLM techniques to a new domain-specific task.
The authors tackled the problem of automating library migration recommendations by proposing LibRec, a framework that combines LLMs with retrieval-augmented generation and in-context learning, achieving evaluation on a benchmark of 2,888 migration records from Python repositories.
In this paper, we propose LibRec, a novel framework that integrates the capabilities of LLMs with retrieval-augmented generation(RAG) techniques to automate the recommendation of alternative libraries. The framework further employs in-context learning to extract migration intents from commit messages to enhance the accuracy of its recommendations. To evaluate the effectiveness of LibRec, we introduce LibEval, a benchmark designed to assess the performance in the library migration recommendation task. LibEval comprises 2,888 migration records associated with 2,368 libraries extracted from 2,324 Python repositories. Each migration record captures source-target library pairs, along with their corresponding migration intents and intent types. Based on LibEval, we evaluated the effectiveness of ten popular LLMs within our framework, conducted an ablation study to examine the contributions of key components within our framework, explored the impact of various prompt strategies on the framework's performance, assessed its effectiveness across various intent types, and performed detailed failure case analyses.