AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study
For CS1 educators, this work provides evidence that AI-generated visualizations can offer context-dependent learning benefits, but highlights the need for personalization.
The study introduced AI-generated animated traces (GATs) to help novice programmers understand program execution, finding selective short-term benefits over textual explanations in CS1 courses (N=1112), with effects moderated by learner engagement.
Introductory programming (CS1) courses often struggle to support students' understanding of program execution. While visualizations can make execution processes explicit, their effectiveness depends on design and context, and empirical evidence for AI-generated visualizations remains limited. We propose Generated Animated Traces (GATs), AI-generated, analogy-based, narrated animations that coordinate source code, execution state, and conceptual analogies. We conduct a study at two institutions in CS1 courses (Python, N=961; Java N=151) comparing GATs to textual explanations. We measure immediate learning performance and experience, end-of-course engagement and exam performance. Results show that GATs can yield selective benefits for immediate learning, but benefits are context-dependent and short-term. We observe that GATs' influence on performance is moderated by learner engagement profiles. This finding underscores the importance of personalized approaches.