QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks
This addresses the problem of generating correct programs from descriptions for developers and AI systems, representing a strong incremental improvement in program synthesis.
The paper tackles program synthesis by introducing QualityFlow, an agentic workflow controlled by LLM quality checks, which achieves state-of-the-art results on benchmarks like MBPP and HumanEval.
We introduce QualityFlow, a dynamic agentic workflow for program synthesis. Given the English description of a programming problem and a set of unit tests, the model's goal is to synthesize the correct program that solves the problem and passes the tests. QualityFlow includes large language model (LLM) agents resembling a software development team, including code generation, testing, and self-debugging. We propose the LLM Quality Checker, which explicitly "imagines" whether the synthesized programs' execution would conform to the unit tests. The Quality Checks dynamically control the workflow, including actions to submit the final answer, clarify the problem statement, and revert previous workflow steps. Our experiments show that the Quality Checker can precisely accept any correct program, mitigate faulty synthesized tests, and prevent potential workflow deviation. QualityFlow establishes the state-of-the-art results on four program synthesis benchmarks: MBPP, HumanEval, and stricter evaluations from MBPP-EvalPlus and HumanEval-EvalPlus.