ConceptKT: A Benchmark for Concept-Level Deficiency Prediction in Knowledge Tracing
This work addresses the need for fine-grained diagnostic feedback in personalized learning, though it is incremental as it builds on existing knowledge tracing methods with a new dataset and evaluation.
The paper tackles the problem of predicting concept-level deficiencies in knowledge tracing, extending beyond binary correctness to identify specific concepts students struggle with, and shows that selecting response histories based on conceptual alignment and semantic similarity improves performance on both correctness prediction and concept-level deficiency identification.
Knowledge Tracing (KT) is a critical technique for modeling student knowledge to support personalized learning. However, most KT systems focus on binary correctness prediction and cannot diagnose the underlying conceptual misunderstandings that lead to errors. Such fine-grained diagnostic feedback is essential for designing targeted instruction and effective remediation. In this work, we introduce the task of concept-level deficiency prediction, which extends traditional KT by identifying the specific concepts a student is likely to struggle with on future problems. We present ConceptKT, a dataset annotated with labels that capture both the concepts required to solve each question and the missing concepts underlying incorrect responses. We investigate in-context learning approaches to KT and evaluate the diagnostic capabilities of various Large Language Models (LLMs) and Large Reasoning Models (LRMs). Different strategies for selecting informative historical records are explored. Experimental results demonstrate that selecting response histories based on conceptual alignment and semantic similarity leads to improved performance on both correctness prediction and concept-level deficiency identification.