Combining physics education and machine learning research to measure evidence of students' mechanistic sensemaking
This work addresses the challenge of automating assessment in science education research, though it is incremental as it builds on existing coding schemes and ML methods.
The researchers tackled the problem of measuring students' mechanistic sensemaking in physics education by developing an ML-based tool that analyzes written responses, achieving useful agreement with human coders and identifying tradeoffs between accuracy and computational expense in different language encoders.
Advances in machine learning (ML) offer new possibilities for science education research. We report on early progress in the design of an ML-based tool to analyze students' mechanistic sensemaking, working from a coding scheme that is aligned with previous work in physics education research (PER) and amenable to recently developed ML classification strategies using language encoders. We describe pilot tests of the tool, in three versions with different language encoders, to analyze sensemaking evident in college students' written responses to brief conceptual questions. The results show, first, that the tool's measurements of sensemaking can achieve useful agreement with a human coder, and, second, that encoder design choices entail a tradeoff between accuracy and computational expense. We discuss the promise and limitations of this approach, providing examples as to how this measurement scheme may serve PER in the future. We conclude with reflections on the use of ML to support PER research, with cautious optimism for strategies of co-design between PER and ML.