Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors
This addresses the need for automated systems to simulate student errors for real-time feedback in education, representing a novel method for a specific bottleneck.
The paper tackles the problem of modeling incorrect reasoning patterns in language models for educational applications, introducing MISTAKE to generate synthetic error examples and showing improved accuracy in simulating student answers, inferring misconceptions, and generating distractors.
Research on reasoning in language models (LMs) predominantly focuses on improving the correctness of their outputs. But some important applications require modeling reasoning patterns that are incorrect. For example, automated systems that can reason about and simulate student errors are useful for providing real-time feedback in the classroom or offline practice for educators-in-training. This paper presents a new method, MISTAKE, that (1) constructs high-quality synthetic examples of reasoning errors by leveraging cycle consistency between incorrect answers and latent misconceptions; and (2) uses the generated data to learn models for student simulation, misconception classification, and answer generation. We evaluate MISTAKE on three educational tasks and find that it results in (1) higher accuracy when simulating incorrect student answers based on specific misconceptions, (2) increased performance inferring latent misconceptions from observed incorrect answers, and (3) higher alignment with expert-written distractor answers when generating incorrect answers (e.g., for multiple-choice tests).