Simulated Students in Tutoring Dialogues: Substance or Illusion?
This addresses the challenge of scalable evaluation for educational technology, though it is incremental as it builds on existing simulation methods.
The paper tackled the problem of evaluating the quality of simulated students in LLM-powered tutoring dialogues, proposing formal definitions and metrics, and found that prompting strategies perform poorly while fine-tuning and optimization yield better but limited results.
Advances in large language models (LLMs) enable many new innovations in education. However, evaluating the effectiveness of new technology requires real students, which is time-consuming and hard to scale up. Therefore, many recent works on LLM-powered tutoring solutions have used simulated students for both training and evaluation, often via simple prompting. Surprisingly, little work has been done to ensure or even measure the quality of simulated students. In this work, we formally define the student simulation task, propose a set of evaluation metrics that span linguistic, behavioral, and cognitive aspects, and benchmark a wide range of student simulation methods on these metrics. We experiment on a real-world math tutoring dialogue dataset, where both automated and human evaluation results show that prompting strategies for student simulation perform poorly; supervised fine-tuning and preference optimization yield much better but still limited performance, motivating future work on this challenging task.