Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions
This addresses the issue of incorrect code generation for developers using LLM assistants, though it is incremental as it builds on existing methods to improve intent inference.
The paper tackles the problem of ambiguous prompts in LLM-based coding assistants by developing a system that asks clarification questions, resulting in a fine-tuned LLM that outperforms zero-shot prompting in generating useful questions and leads to more accurate code responses in user studies.
Large Language Models (LLMs) are increasingly used as coding assistants. However, the ambiguity of the developer's prompt often leads to incorrect code generation, as current models struggle to infer user intent without extensive prompt engineering or external context. This work aims to build an LLM-based coding assistant that mimics the human code review process by asking clarification questions when faced with ambiguous or under-specified queries. Our end-to-end system includes (1) a query classifier trained to detect unclear programming-related queries and (2) a fine-tuned LLM that generates clarification questions. Our evaluation shows that the fine-tuned LLM outperforms standard zero-shot prompting in generating useful clarification questions. Furthermore, our user study indicates that users find the clarification questions generated by our model to outperform the baseline, demonstrating that our coding assistant produces more accurate and helpful code responses compared to baseline coding assistants.