PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation
This addresses the challenge of enhancing code generation quality in underrepresented programming languages, offering a robust solution for developers and AI systems, though it is incremental as it builds on existing retrieval-augmented generation methods.
The paper tackles the problem of selecting effective few-shot examples for code generation with large language models, particularly for underrepresented programming languages, by proposing PERC, a framework that retrieves examples based on algorithmic plans, which improves generation accuracy and outperforms state-of-the-art methods on benchmarks like CodeContests, HumanEval, and MultiPL-E.
Code generation with large language models has shown significant promise, especially when employing retrieval-augmented generation (RAG) with few-shot examples. However, selecting effective examples that enhance generation quality remains a challenging task, particularly when the target programming language (PL) is underrepresented. In this study, we present two key findings: (1) retrieving examples whose presented algorithmic plans can be referenced for generating the desired behavior significantly improves generation accuracy, and (2) converting code into pseudocode effectively captures such algorithmic plans, enhancing retrieval quality even when the source and the target PLs are different. Based on these findings, we propose Plan-as-query Example Retrieval for few-shot prompting in Code generation (PERC), a novel framework that utilizes algorithmic plans to identify and retrieve effective examples. We validate the effectiveness of PERC through extensive experiments on the CodeContests, HumanEval and MultiPL-E benchmarks: PERC consistently outperforms the state-of-the-art RAG methods in code generation, both when the source and target programming languages match or differ, highlighting its adaptability and robustness in diverse coding environments.