Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?
This work addresses the challenge of designing effective AI-assisted learning tools for novice programming students, though it is incremental as it builds on prior research.
The study tackled the problem of how students interact with and request feedback from a GenAI chatbot in programming education, finding that 75% of chatbot responses aligned with requested feedback types and students' requests followed a specific sequence.
Building on prior research on Generative AI (GenAI) and related tools for programming education, we developed SCRIPT, a chatbot based on ChatGPT-4o-mini, to support novice learners. SCRIPT allows for open-ended interactions and structured guidance through predefined prompts. We evaluated the tool via an experiment with 136 students from an introductory programming course at a large German university and analyzed how students interacted with SCRIPT while solving programming tasks with a focus on their feedback preferences. The results reveal that students' feedback requests seem to follow a specific sequence. Moreover, the chatbot responses aligned well with students' requested feedback types (in 75%), and it adhered to the system prompt constraints. These insights inform the design of GenAI-based learning support systems and highlight challenges in balancing guidance and flexibility in AI-assisted tools.