Your Students Don't Use LLMs Like You Wish They Did
For researchers building educational dialogue systems, this work provides automated metrics to detect misalignment between pedagogical goals and actual student usage, revealing that current evaluation methods miss critical turn-by-turn patterns.
The paper introduces six computational metrics to evaluate pedagogical alignment in student-AI dialogue, analyzing 12,650 messages across 500 conversations. It finds that students primarily use conversational tutors for answer-extraction rather than sustained learning, with deployment context being the strongest predictor of usage patterns.
Educational NLP systems are typically evaluated using engagement metrics and satisfaction surveys, which are at best a proxy for meeting pedagogical goals. We introduce six computational metrics for automated evaluation of pedagogical alignment in student-AI dialogue. We validate our metrics through analysis of 12,650 messages across 500 conversations from four courses. Using our metrics, we identify a fundamental misalignment: educators design conversational tutors for sustained learning dialogue, but students mainly use them for answer-extraction. Deployment context is the strongest predictor of usage patterns, outweighing student preference or system design: when AI tools are optional, usage concentrates around deadlines; when integrated into course structure, students ask for solutions to verbatim assignment questions. Whole-dialogue evaluation misses these turn-by-turn patterns. Our metrics will enable researchers building educational dialogue systems to measure whether they are achieving their pedagogical goals.