Large-scale Cloze Test Dataset Created by Teachers
This dataset addresses the need for more challenging and nuanced language evaluation tools in education and NLP, though it is incremental as it builds on existing cloze test methods.
The authors introduced CLOTH, the first large-scale human-created cloze test dataset from middle-school and high-school exams, designed to require deeper language understanding than automated datasets. They found that baseline models, including one trained on the One Billion Word Corpus, significantly underperformed humans, with the key bottleneck identified as limited long-term context comprehension.
Cloze tests are widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH, containing questions used in middle-school and high-school language exams. With missing blanks carefully created by teachers and candidate choices purposely designed to be nuanced, CLOTH requires a deeper language understanding and a wider attention span than previously automatically-generated cloze datasets. We test the performance of dedicatedly designed baseline models including a language model trained on the One Billion Word Corpus and show humans outperform them by a significant margin. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending the long-term context to be the key bottleneck.