Effects of Human vs. Automatic Feedback on Students' Understanding of AI Concepts and Programming Style
This study provides evidence for educators and course designers that human feedback, especially regarding the syntax-logic relationship, can improve conceptual understanding and overall performance for students in programming-heavy AI courses.
This paper investigated the impact of human versus automatic feedback on 90 students in an introductory AI programming course. Students receiving human-written feedback, which included syntax-logic relations and style recommendations, demonstrated a better conceptual understanding and achieved higher overall course grades, particularly in the middle two quartiles.
The use of automatic grading tools has become nearly ubiquitous in large undergraduate programming courses, and recent work has focused on improving the quality of automatically generated feedback. However, there is a relative lack of data directly comparing student outcomes when receiving computer-generated feedback and human-written feedback. This paper addresses this gap by splitting one 90-student class into two feedback groups and analyzing differences in the two cohorts' performance. The class is an intro to AI with programming HW assignments. One group of students received detailed computer-generated feedback on their programming assignments describing which parts of the algorithms' logic was missing; the other group additionally received human-written feedback describing how their programs' syntax relates to issues with their logic, and qualitative (style) recommendations for improving their code. Results on quizzes and exam questions suggest that human feedback helps students obtain a better conceptual understanding, but analyses found no difference between the groups' ability to collaborate on the final project. The course grade distribution revealed that students who received human-written feedback performed better overall; this effect was the most pronounced in the middle two quartiles of each group. These results suggest that feedback about the syntax-logic relation may be a primary mechanism by which human feedback improves student outcomes.