Controlling Cloze-test Question Item Difficulty with PLM-based Surrogate Models for IRT Assessment
This work addresses the need for adaptive testing in education by enabling automated generation of varied-difficulty questions, though it is incremental as it builds on existing PLM and IRT methods.
The authors tackled the problem of generating multiple-choice cloze test questions with controlled difficulty levels by using pre-trained language models as surrogate models for item response theory assessment, achieving effective control and evaluation of difficulty without human test subjects.
Item difficulty plays a crucial role in adaptive testing. However, few works have focused on generating questions of varying difficulty levels, especially for multiple-choice (MC) cloze tests. We propose training pre-trained language models (PLMs) as surrogate models to enable item response theory (IRT) assessment, avoiding the need for human test subjects. We also propose two strategies to control the difficulty levels of both the gaps and the distractors using ranking rules to reduce invalid distractors. Experimentation on a benchmark dataset demonstrates that our proposed framework and methods can effectively control and evaluate the difficulty levels of MC cloze tests.