Towards Mining Effective Pedagogical Strategies from Learner-LLM Educational Dialogues
This work addresses the gap in evaluating LLM-based educational applications by focusing on dialogue dynamics, which is important for educators and developers, but it is incremental as it builds on existing dialogue analysis approaches.
The paper tackles the problem of evaluating large language models in education by analyzing learner-LLM dialogues to identify effective pedagogical strategies, using methods like dialogue act annotation and pattern mining, with early insights presented as a foundation for future research.
Dialogue plays a crucial role in educational settings, yet existing evaluation methods for educational applications of large language models (LLMs) primarily focus on technical performance or learning outcomes, often neglecting attention to learner-LLM interactions. To narrow this gap, this AIED Doctoral Consortium paper presents an ongoing study employing a dialogue analysis approach to identify effective pedagogical strategies from learner-LLM dialogues. The proposed approach involves dialogue data collection, dialogue act (DA) annotation, DA pattern mining, and predictive model building. Early insights are outlined as an initial step toward future research. The work underscores the need to evaluate LLM-based educational applications by focusing on dialogue dynamics and pedagogical strategies.