Evaluating Learner Representations for Differentiation Prior to Instructional Outcomes
This provides a practical pre-deployment criterion for educational AI systems to assess representation quality for personalization, though it is incremental as it builds on existing representation evaluation methods.
This work tackles the problem of evaluating learner representations in educational AI when instructional outcomes are unavailable, by introducing distinctiveness as a measure to assess separation between learners. Results show that learner-level representations yield higher separation and more reliable discrimination than interaction-level representations, with concrete improvements in clustering structure and pairwise discrimination.
Learner representations play a central role in educational AI systems, yet it is often unclear whether they preserve meaningful differences between students when instructional outcomes are unavailable or highly context-dependent. This work examines how to evaluate learner representations based on whether they retain separation between learners under a shared comparison rule. We introduce distinctiveness, a representation-level measure that evaluates how each learner differs from others in the cohort using pairwise distances, without requiring clustering, labels, or task-specific evaluation. Using student-authored questions collected through a conversational AI agent in an online learning environment, we compare representations based on individual questions with representations that aggregate patterns across a student's interactions over time. Results show that learner-level representations yield higher separation, stronger clustering structure, and more reliable pairwise discrimination than interaction-level representations. These findings demonstrate that learner representations can be evaluated independently of instructional outcomes and provide a practical pre-deployment criterion using distinctiveness as a diagnostic metric for assessing whether a representation supports differentiated modeling or personalization.