Can Transformer Language Models Predict Psychometric Properties?
This work addresses the problem of understanding and leveraging language models for psychometric applications, offering incremental insights into their alignment with human cognitive processes.
The study investigated whether transformer-based language models can predict psychometric properties of test items, such as difficulty and discrimination, by comparing predictions from human and LM responses on a linguistic competency test. It found that transformer LMs predicted properties well in some categories but poorly in others, revealing insights into human-LM reasoning differences.
Transformer-based language models (LMs) continue to advance state-of-the-art performance on NLP benchmark tasks, including tasks designed to mimic human-inspired "commonsense" competencies. To better understand the degree to which LMs can be said to have certain linguistic reasoning skills, researchers are beginning to adapt the tools and concepts of the field of psychometrics. But to what extent can the benefits flow in the other direction? I.e., can LMs be of use in predicting what the psychometric properties of test items will be when those items are given to human participants? We gather responses from numerous human participants and LMs (transformer and non-transformer-based) on a broad diagnostic test of linguistic competencies. We then use the responses to calculate standard psychometric properties of the items in the diagnostic test, using the human responses and the LM responses separately. We then determine how well these two sets of predictions match. We find cases in which transformer-based LMs predict psychometric properties consistently well in certain categories but consistently poorly in others, thus providing new insights into fundamental similarities and differences between human and LM reasoning.