Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?
This addresses the problem of accelerating educational test development by reducing reliance on human pilot studies, but it is incremental as it shows current limitations.
The study evaluated whether large language models (LLMs) produce human-like responses to multiple-choice test items in reading, U.S. history, and economics, finding that while calibration improves plausibility, correlations with human responses are weak, making them unsuitable for zero-shot use in test development.
Knowing how test takers answer items in educational assessments is essential for test development, to evaluate item quality, and to improve test validity. However, this process usually requires extensive pilot studies with human participants. If large language models (LLMs) exhibit human-like response behavior to test items, this could open up the possibility of using them as pilot participants to accelerate test development. In this paper, we evaluate the human-likeness or psychometric plausibility of responses from 18 instruction-tuned LLMs with two publicly available datasets of multiple-choice test items across three subjects: reading, U.S. history, and economics. Our methodology builds on two theoretical frameworks from psychometrics which are commonly used in educational assessment, classical test theory and item response theory. The results show that while larger models are excessively confident, their response distributions can be more human-like when calibrated with temperature scaling. In addition, we find that LLMs tend to correlate better with humans in reading comprehension items compared to other subjects. However, the correlations are not very strong overall, indicating that LLMs should not be used for piloting educational assessments in a zero-shot setting.