Understanding Student Perceptions, Mistakes, and Debugging Approaches when Solving Natural Language Programming Tasks
For CS educators, this work identifies specific novice mistakes and debugging strategies in natural-language programming, informing pedagogical approaches for teaching prompt engineering.
This study analyzed over 900 CS1 students solving Prompt Problems, finding that students generally found prompting easier and more enjoyable than traditional coding, but common mistakes involved omitting key details, and when prompts failed, students focused more on clarifying intent than debugging code.
Learning to communicate with code-generating AI models is an emerging skill for novice programmers. One recent pedagogical approach, Prompt Problems, has students solve computational tasks by writing natural-language prompts for code-generating AI models. However, little is known about the specific prompt-level mistakes novice programmers make, the kinds of computational details they fail to communicate, and what strategies they use to recover when generated code is incorrect. In a CS1 course, we studied attempts by more than 900 students to solve dialogue-based Prompt Problems. We analyzed student reflections, unsuccessful prompts, and reported debugging strategies. Compared to traditional coding tasks, students generally found prompting easier, more enjoyable, and better targeted at developing problem-solving skills. The most common mistakes are related to the omission of key details, suggesting both a failure to acknowledge their importance and over-reliance on AI to infer them. When prompts failed, students focused more on clarifying their intent and reflecting on the provided problem details than on tracing generated code or examining test cases.