GenX: Mastering Code and Test Generation with Execution Feedback
This addresses the challenge of limited test case availability in code generation for developers, though it is incremental as it builds on existing methods with novel strategies.
The paper tackles the problem of generating code and tests without relying on pre-existing test cases by proposing a concurrent training approach for code and test generation models using execution feedback, resulting in models that outperform those trained on the original dataset when iteratively trained with more test cases and code solutions.
Recent advancements in language modeling have enabled the translation of natural language into code, and the use of execution feedback to improve code generation. However, these methods often rely heavily on pre-existing test cases, which may not always be available or comprehensive. In this work, we propose a novel approach that concurrently trains a code generation model and a test generation model, utilizing execution feedback to refine and enhance the performance of both. We introduce two strategies for test and code data augmentation and a new scoring function for code and test ranking. We experiment on the APPS dataset and demonstrate that our approach can effectively generate and augment test cases, filter and synthesize correct code solutions, and rank the quality of generated code and tests. The results demonstrate that our models, when iteratively trained with an increasing number of test cases and code solutions, outperform those trained on the original dataset.