Selective Prompt Anchoring for Code Generation
This addresses code generation errors for software developers, offering an incremental improvement over existing methods.
The paper tackles the problem of large language models (LLMs) generating incorrect code due to attention dilution, where models pay less attention to user prompts as more code is produced, and proposes Selective Prompt Anchoring (SPA) to guide LLMs to focus more on user intent, resulting in up to a 12.9% improvement in Pass@1 across six benchmarks.
Recent advances in large language models (LLMs) have transformed software development by automatically generating code from natural language. Yet challenges remain in generating fully correct code that aligns with user intent. Our study reveals that LLMs tend to pay less attention to user prompts as more code tokens are generated. We hypothesize that this attention dilution issue is an important reason for code generation errors. To mitigate this issue, we propose Selective Prompt Anchoring (SPA) to guide code LLMs to pay more attention to user intent when generating code. We evaluate SPA using six base LLMs across six benchmarks. Our results demonstrate that SPA enhances Pass@1 by up to 12.9%, consistently outperforming SOTA code generation methods in all settings. Our code is available at https://github.com/magic-YuanTian/Selective-Prompt-Anchoring.