Large Language Models as Students Who Think Aloud: Overly Coherent, Verbose, and Confident
This work addresses the problem of accurately modeling novice reasoning for educational AI systems, but it is incremental as it builds on existing evaluations by focusing on reasoning quality rather than just accuracy.
The study evaluated large language models (LLMs) as novices in AI-based tutoring systems by comparing their reasoning to human think-alouds from chemistry problems, finding that GPT-4.1 generated overly coherent, verbose, and less variable reasoning while consistently overestimating learner performance.
Large language models (LLMs) are increasingly embedded in AI-based tutoring systems. Can they faithfully model novice reasoning and metacognitive judgments? Existing evaluations emphasize problem-solving accuracy, overlooking the fragmented and imperfect reasoning that characterizes human learning. We evaluate LLMs as novices using 630 think-aloud utterances from multi-step chemistry tutoring problems with problem-solving logs of student hint use, attempts, and problem context. We compare LLM-generated reasoning to human learner utterances under minimal and extended contextual prompting, and assess the models' ability to predict step-level learner success. Although GPT-4.1 generates fluent and contextually appropriate continuations, its reasoning is systematically over-coherent, verbose, and less variable than human think-alouds. These effects intensify with a richer problem-solving context during prompting. Learner performance was consistently overestimated. These findings highlight epistemic limitations of simulating learning with LLMs. We attribute these limitations to LLM training data, including expert-like solutions devoid of expressions of affect and working memory constraints during problem solving. Our evaluation framework can guide future design of adaptive systems that more faithfully support novice learning and self-regulation using generative artificial intelligence.