Measuring Five Accountable Talk Moves to Improve Instruction at Scale
This work addresses the challenge of improving instruction for novice online teachers through automated feedback, though it is incremental as it builds on existing accountable talk theory and applies known methods to new data.
The researchers tackled the problem of providing scalable feedback on teacher instruction by fine-tuning RoBERTa and GPT models to identify five accountable talk moves from transcripts, finding that GPT-3 had higher precision but variable recall, and correlating talk move usage with student outcomes showed positive effects, with connecting student ideas having the largest impact.
Providing consistent, individualized feedback to teachers on their instruction can improve student learning outcomes. Such feedback can especially benefit novice instructors who teach on online platforms and have limited access to instructional training. To build scalable measures of instruction, we fine-tune RoBERTa and GPT models to identify five instructional talk moves inspired by accountable talk theory: adding on, connecting, eliciting, probing and revoicing students' ideas. We fine-tune these models on a newly annotated dataset of 2500 instructor utterances derived from transcripts of small group instruction in an online computer science course, Code in Place. Although we find that GPT-3 consistently outperforms RoBERTa in terms of precision, its recall varies significantly. We correlate the instructors' use of each talk move with indicators of student engagement and satisfaction, including students' section attendance, section ratings, and assignment completion rates. We find that using talk moves generally correlates positively with student outcomes, and connecting student ideas has the largest positive impact. These results corroborate previous research on the effectiveness of accountable talk moves and provide exciting avenues for using these models to provide instructors with useful, scalable feedback.