The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance
For educators and developers of automated tutoring systems, this work provides evidence that LLM-generated feedback can improve efficiency in programming assignments, though the effects are incremental and context-specific.
This study investigates how LLM-generated feedback with varying levels of guidance affects programming students' problem-solving behavior, finding that such feedback reduces time to solution compared to compiler-only feedback, with less guided feedback yielding slightly stronger effects.
When programming students encounter errors in their code, compiler messages or static analysis output often provide limited guidance, particularly for novice programmers. Personalized feedback from instructors can be effective but does not scale well. Recent advances in large language models (LLMs) enable automated feedback generation at scale. This study examines whether LLM-generated feedback with different levels of guidance is associated with differences in students' problem-solving behavior. We analyze effects on time to solution and number of attempts, and examine whether these effects differ by programming experience. We design three feedback types and compare them to a baseline in which students receive only compiler error messages. Results from an online programming course show that LLM-generated feedback is associated with faster time to solution compared to the no-feedback baseline, with less guided feedback showing slightly stronger effects. Overall, the findings suggest that feedback structure plays an important role in how students progress toward correct solutions and motivate further work on adaptive feedback designs and longer-term learning outcomes.