Enhancing Programming Error Messages in Real Time with Generative AI
This addresses the long-standing problem of cryptic programming error messages for computer science students and instructors, though it appears incremental by testing an existing AI method in a new educational context.
The researchers investigated whether adding generative AI (GPT-4) to an automated assessment tool (Athene) improves programming error messages for students, finding that interface design critically affects usability rather than automatically enhancing feedback quality.
Generative AI is changing the way that many disciplines are taught, including computer science. Researchers have shown that generative AI tools are capable of solving programming problems, writing extensive blocks of code, and explaining complex code in simple terms. Particular promise has been shown in using generative AI to enhance programming error messages. Both students and instructors have complained for decades that these messages are often cryptic and difficult to understand. Yet recent work has shown that students make fewer repeated errors when enhanced via GPT-4. We extend this work by implementing feedback from ChatGPT for all programs submitted to our automated assessment tool, Athene, providing help for compiler, run-time, and logic errors. Our results indicate that adding generative AI to an automated assessment tool does not necessarily make it better and that design of the interface matters greatly to the usability of the feedback that GPT-4 provided.