Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators
This work addresses the need for scalable and cost-effective survey item validation in psychometrics, particularly for LLM assessment, by introducing a novel simulation-based approach.
The paper tackles the problem of validating psychometric survey items for large language models (LLMs) by proposing a framework that simulates virtual respondents with trait-response mediators to identify items that robustly measure intended traits, with experiments on three psychological trait theories showing effective identification of high-validity items.
As psychometric surveys are increasingly used to assess the traits of large language models (LLMs), the need for scalable survey item generation suited for LLMs has also grown. A critical challenge here is ensuring the construct validity of generated items, i.e., whether they truly measure the intended trait. Traditionally, this requires costly, large-scale human data collection. To make it efficient, we present a framework for virtual respondent simulation using LLMs. Our central idea is to account for mediators: factors through which the same trait can give rise to varying responses to a survey item. By simulating respondents with diverse mediators, we identify survey items that robustly measure intended traits. Experiments on three psychological trait theories (Big5, Schwartz, VIA) show that our mediator generation methods and simulation framework effectively identify high-validity items. LLMs demonstrate the ability to generate plausible mediators from trait definitions and to simulate respondent behavior for item validation. Our problem formulation, metrics, methodology, and dataset open a new direction for cost-effective survey development and a deeper understanding of how LLMs simulate human survey responses. We publicly release our dataset and code to support future work.