Prompting LLMs for Code Editing: Struggles and Remedies
This addresses a practical issue for software engineers using LLM-powered coding assistants, though it is incremental as it builds on existing tools.
The paper tackled the problem of developers struggling to effectively prompt LLMs for code editing by analyzing usage logs and identifying missing information in prompts, resulting in a 27% improvement in edit correctness with their proposed tool AutoPrompter.
Large Language Models (LLMs) are rapidly transforming software engineering, with coding assistants embedded in an IDE becoming increasingly prevalent. While research has focused on improving the tools and understanding developer perceptions, a critical gap exists in understanding how developers actually use these tools in their daily workflows, and, crucially, where they struggle. This paper addresses part of this gap through a multi-phased investigation of developer interactions with an LLM-powered code editing and transformation feature, Transform Code, in an IDE widely used at Google. First, we analyze telemetry logs of the feature usage, revealing that frequent re-prompting can be an indicator of developer struggles with using Transform Code. Second, we conduct a qualitative analysis of unsatisfactory requests, identifying five key categories of information often missing from developer prompts. Finally, based on these findings, we propose and evaluate a tool, AutoPrompter, for automatically improving prompts by inferring missing information from the surrounding code context, leading to a 27% improvement in edit correctness on our test set.