JumpCoder: Go Beyond Autoregressive Coder via Online Modification
This addresses a key problem for developers and AI researchers by enhancing code generation accuracy and flexibility, though it is an incremental improvement over existing methods.
The paper tackles the limitation of autoregressive code generation in large language models, which lacks reversibility and leads to error propagation, by introducing JumpCoder, a model-agnostic framework that enables online modification and non-sequential generation, resulting in significant improvements across multiple benchmarks with six state-of-the-art models.
While existing code large language models (code LLMs) exhibit impressive capabilities in code generation, their autoregressive sequential generation inherently lacks reversibility. This limitation hinders them from timely correcting previous missing statements during coding as humans do, often leading to error propagation and suboptimal performance. We introduce JumpCoder, a novel model-agnostic framework that enables human-like online modification and non-sequential generation to augment code LLMs. The key idea behind JumpCoder is to insert new code into the currently generated code when necessary during generation, which is achieved through an auxiliary infilling model that works in tandem with the code LLM. Since identifying the best infill position beforehand is intractable, we adopt an \textit{infill-first, judge-later} strategy, which experiments with filling at the $k$ most critical positions following the generation of each line, and uses an Abstract Syntax Tree (AST) parser alongside the Generation Model Scoring to effectively judge the validity of each potential infill. Extensive experiments using six state-of-the-art code LLMs across multiple and multilingual benchmarks consistently indicate significant improvements over all baselines. Our code is public at https://github.com/Keytoyze/JumpCoder.