Exploring LLMs for Predicting Tutor Strategy and Student Outcomes in Dialogues
This addresses the problem of improving AI tutoring systems for online learning, but it is incremental as it highlights limitations without proposing new solutions.
The study tackled predicting tutor strategies and student outcomes in math tutoring dialogues using LLMs like Llama 3 and GPT-4o, finding that even state-of-the-art models struggle with strategy prediction, though strategies strongly indicate outcomes.
Tutoring dialogues have gained significant attention in recent years, given the prominence of online learning and the emerging tutoring abilities of artificial intelligence (AI) agents powered by large language models (LLMs). Recent studies have shown that the strategies used by tutors can have significant effects on student outcomes, necessitating methods to predict how tutors will behave and how their actions impact students. However, few works have studied predicting tutor strategy in dialogues. Therefore, in this work we investigate the ability of modern LLMs, particularly Llama 3 and GPT-4o, to predict both future tutor moves and student outcomes in dialogues, using two math tutoring dialogue datasets. We find that even state-of-the-art LLMs struggle to predict future tutor strategy while tutor strategy is highly indicative of student outcomes, outlining a need for more powerful methods to approach this task.