QG-SMS: Enhancing Test Item Analysis via Student Modeling and Simulation
This addresses the need for better evaluation methods in educational technology for educators and researchers, though it is incremental as it builds on existing QG tasks by incorporating test item analysis.
The paper tackled the problem of evaluating question generation (QG) in educational assessments by introducing test item analysis to assess question quality across dimensions like difficulty and discrimination. The result was the QG-SMS framework, which uses large language models for student modeling and simulation, showing more effective and robust assessment in experiments and human evaluations.
While the Question Generation (QG) task has been increasingly adopted in educational assessments, its evaluation remains limited by approaches that lack a clear connection to the educational values of test items. In this work, we introduce test item analysis, a method frequently used by educators to assess test question quality, into QG evaluation. Specifically, we construct pairs of candidate questions that differ in quality across dimensions such as topic coverage, item difficulty, item discrimination, and distractor efficiency. We then examine whether existing QG evaluation approaches can effectively distinguish these differences. Our findings reveal significant shortcomings in these approaches with respect to accurately assessing test item quality in relation to student performance. To address this gap, we propose a novel QG evaluation framework, QG-SMS, which leverages Large Language Model for Student Modeling and Simulation to perform test item analysis. As demonstrated in our extensive experiments and human evaluation study, the additional perspectives introduced by the simulated student profiles lead to a more effective and robust assessment of test items.