Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods
This addresses the issue of evaluating student-generated questions for educators, but it is incremental as it builds on existing automated assessment methods.
The study tackled the problem of automatically assessing multiple-choice questions for item-writing flaws, comparing a rule-based method and GPT-4, and found that the rule-based method correctly detected 91% of flaws versus 79% for GPT-4 on 200 student-generated questions.
Multiple-choice questions with item-writing flaws can negatively impact student learning and skew analytics. These flaws are often present in student-generated questions, making it difficult to assess their quality and suitability for classroom usage. Existing methods for evaluating multiple-choice questions often focus on machine readability metrics, without considering their intended use within course materials and their pedagogical implications. In this study, we compared the performance of a rule-based method we developed to a machine-learning based method utilizing GPT-4 for the task of automatically assessing multiple-choice questions based on 19 common item-writing flaws. By analyzing 200 student-generated questions from four different subject areas, we found that the rule-based method correctly detected 91% of the flaws identified by human annotators, as compared to 79% by GPT-4. We demonstrated the effectiveness of the two methods in identifying common item-writing flaws present in the student-generated questions across different subject areas. The rule-based method can accurately and efficiently evaluate multiple-choice questions from multiple domains, outperforming GPT-4 and going beyond existing metrics that do not account for the educational use of such questions. Finally, we discuss the potential for using these automated methods to improve the quality of questions based on the identified flaws.