Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?
This work addresses the interpretability and trust gap for programmers using LLMs for code generation, though it is incremental in analyzing attention patterns rather than proposing new methods.
The study investigated whether large language models (LLMs) attend to the same parts of task descriptions as human programmers during code generation, finding a consistent misalignment across six LLMs on two benchmarks, with manual analysis of 211 incorrect snippets revealing five attention patterns that explain many errors.
Large Language Models (LLMs) have recently been widely used for code generation. Due to the complexity and opacity of LLMs, little is known about how these models generate code. We made the first attempt to bridge this knowledge gap by investigating whether LLMs attend to the same parts of a task description as human programmers during code generation. An analysis of six LLMs, including GPT-4, on two popular code generation benchmarks revealed a consistent misalignment between LLMs' and programmers' attention. We manually analyzed 211 incorrect code snippets and found five attention patterns that can be used to explain many code generation errors. Finally, a user study showed that model attention computed by a perturbation-based method is often favored by human programmers. Our findings highlight the need for human-aligned LLMs for better interpretability and programmer trust.