Evaluating Large Language Models in Analysing Classroom Dialogue
This work addresses the problem of efficient teaching diagnosis and quality improvement for educational researchers, but it is incremental as it applies an existing method to a new domain.
This study tackled the problem of analyzing classroom dialogue, which is traditionally labor-intensive, by applying GPT-4 to datasets from middle school mathematics and Chinese classes, resulting in substantial time savings and high consistency with human coders, though with some discrepancies in specific codes.
This study explores the application of Large Language Models (LLMs), specifically GPT-4, in the analysis of classroom dialogue, a crucial research task for both teaching diagnosis and quality improvement. Recognizing the knowledge-intensive and labor-intensive nature of traditional qualitative methods in educational research, this study investigates the potential of LLM to streamline and enhance the analysis process. The study involves datasets from a middle school, encompassing classroom dialogues across mathematics and Chinese classes. These dialogues were manually coded by educational experts and then analyzed using a customised GPT-4 model. This study focuses on comparing manual annotations with the outputs of GPT-4 to evaluate its efficacy in analyzing educational dialogues. Time efficiency, inter-coder agreement, and inter-coder reliability between human coders and GPT-4 are evaluated. Results indicate substantial time savings with GPT-4, and a high degree of consistency in coding between the model and human coders, with some discrepancies in specific codes. These findings highlight the strong potential of LLM in teaching evaluation and facilitation.