Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation
This addresses the need for more secure and efficient code generation in AI-assisted programming, representing a novel approach rather than an incremental improvement.
The paper tackles the problem of secure code generation by proposing a paradigm shift from monotonic token generation to a dynamic, self-correcting process, resulting in a significant reduction in vulnerabilities with minimal inference overhead.
Large Language Model (LLM) based code generation is predominantly formulated as a strictly monotonic process, appending tokens linearly to an immutable prefix. This formulation contrasts to the cognitive process of programming, which is inherently interleaved with forward generation and on-the-fly revision. While prior works attempt to introduce revision via post-hoc agents or external static tools, they either suffer from high latency or fail to leverage the model's intrinsic semantic reasoning. In this paper, we propose Stream of Revision, a paradigm shift that elevates code generation from a monotonic stream to a dynamic, self-correcting trajectory by leveraging model's intrinsic capabilities. We introduce specific action tokens that enable the model to seamlessly backtrack and edit its own history within a single forward pass. By internalizing the revision loop, our framework Stream of Revision allows the model to activate its latent capabilities just-in-time without external dependencies. Empirical results on secure code generation show that Stream of Revision significantly reduces vulnerabilities with minimal inference overhead.