AutoTest: Evolutionary Code Solution Selection with Test Cases
This addresses the challenge of accurate code solution selection for developers using code generation tools, though it is incremental as it builds on existing methods.
The study tackled the problem of selecting correct code solutions from multiple candidates by proposing AutoTest, which combines automated test case generation with an evolutionary genetic algorithm, achieving a 10% improvement in pass@1 score on the HumanEval benchmark.
With the development of code generation techniques, selecting the correct code solution from multiple candidate solutions has become a crucial task. This study proposes AutoTest, a novel technique that combines automated test case generation with code solution execution to optimize the selection process using an evolutionary genetic algorithm. Firstly, AutoTest utilizes large pre-trained language models such as codegen-16B, code-davinci-002, and incoder-6B to provide code solutions and their corresponding test cases. Then, by executing the code solutions and evaluating their performance on the test cases, a consensus set is formed. Fine-grained ranking is achieved through the selection, mutation, and crossover mechanisms based on the evolutionary genetic algorithm, with the adjustment of alpha and beta parameters. Finally, the best code solution is chosen. AutoTest demonstrates significant performance improvements on the HumanEval benchmark test. The HumanEval dataset consists of 164 programming problems, and AutoTest achieves approximately a 10% improvement over the baseline method in terms of pass@1 score.