Training Optimus Prime, M.D.: Generating Medical Certification Items by Fine-Tuning OpenAI's gpt2 Transformer Model
This addresses the problem of automating test item creation for medical certification, though it is incremental as it builds on existing transformer models.
The authors tackled automated item generation for medical certification by fine-tuning OpenAI's gpt2 on PubMed articles to generate case vignettes and distractors for multiple-choice items, showing promise in producing draft text for human item writers.
This article describes new results of an application using transformer-based language models to automated item generation (AIG), an area of ongoing interest in the domain of certification testing as well as in educational measurement and psychological testing. OpenAI's gpt2 pre-trained 345M parameter language model was retrained using the public domain text mining set of PubMed articles and subsequently used to generate item stems (case vignettes) as well as distractor proposals for multiple-choice items. This case study shows promise and produces draft text that can be used by human item writers as input for authoring. Future experiments with more recent transformer models (such as Grover, TransformerXL) using existing item pools are expected to improve results further and to facilitate the development of assessment materials.