Field-testing items using artificial intelligence: Natural language processing with transformers
This work addresses the problem of efficiently field-testing educational items for test developers, though it is incremental as it applies an existing AI method to a new domain.
The researchers used 5,000 RoBERTa model variations to take an English literacy exam with 29 multiple-choice questions, finding that the AI-generated psychometric properties of the items showed some agreement with those from human examinee data.
Five thousand variations of the RoBERTa model, an artificially intelligent "transformer" that can understand text language, completed an English literacy exam with 29 multiple-choice questions. Data were used to calculate the psychometric properties of the items, which showed some degree of agreement to those obtained from human examinee data.