Who's the Best Detective? LLMs vs. MLs in Detecting Incoherent Fourth Grade Math Answers
This work addresses the time-consuming task for teachers of reviewing student answers, but it is incremental as it compares existing methods on new data without introducing novel solutions.
The study tackled the problem of automatically detecting incoherent fourth-grade math answers to assist teachers, finding that Large Language Models (LLMs) performed worse than traditional Machine Learning (ML) classifiers, with specific challenges in handling recursive questions and student misspellings.
Written answers to open-ended questions can have a higher long-term effect on learning than multiple-choice questions. However, it is critical that teachers immediately review the answers, and ask to redo those that are incoherent. This can be a difficult task and can be time-consuming for teachers. A possible solution is to automate the detection of incoherent answers. One option is to automate the review with Large Language Models (LLM). In this paper, we analyze the responses of fourth graders in mathematics using three LLMs: GPT-3, BLOOM, and YOU. We used them with zero, one, two, three and four shots. We compared their performance with the results of various classifiers trained with Machine Learning (ML). We found that LLMs perform worse than MLs in detecting incoherent answers. The difficulty seems to reside in recursive questions that contain both questions and answers, and in responses from students with typical fourth-grader misspellings. Upon closer examination, we have found that the ChatGPT model faces the same challenges.