Test Case-Informed Knowledge Tracing for Open-ended Coding Tasks
This addresses the problem of fine-grained student modeling for educators in computer science education, though it is incremental as it builds on existing knowledge tracing and multi-task learning approaches.
The paper tackles the challenge of modeling student knowledge in open-ended coding tasks by introducing TIKTOC, a framework that simultaneously predicts test case pass/fail outcomes and student code, outperforming existing knowledge tracing methods that rely only on overall scores.
Open-ended coding tasks, which ask students to construct programs according to certain specifications, are common in computer science education. Student modeling can be challenging since their open-ended nature means that student code can be diverse. Traditional knowledge tracing (KT) models that only analyze response correctness may not fully capture nuances in student knowledge from student code. In this paper, we introduce Test case-Informed Knowledge Tracing for Open-ended Coding (TIKTOC), a framework to simultaneously analyze and predict both open-ended student code and whether the code passes each test case. We augment the existing CodeWorkout dataset with the test cases used for a subset of the open-ended coding questions, and propose a multi-task learning KT method to simultaneously analyze and predict 1) whether a student's code submission passes each test case and 2) the student's open-ended code, using a large language model as the backbone. We quantitatively show that these methods outperform existing KT methods for coding that only use the overall score a code submission receives. We also qualitatively demonstrate how test case information, combined with open-ended code, helps us gain fine-grained insights into student knowledge.