Comparing Human Expertise and Large Language Models Embeddings in Content Validity Assessment of Personality Tests
This work addresses the problem of scalable and objective content validity assessment for psychometric test developers, offering a hybrid approach that is incremental in combining existing methods.
The study compared human experts and large language models (LLMs) in assessing content validity of personality tests (BFQ and BFI), finding that humans excelled with behaviorally rich BFQ items while LLMs performed better with linguistically concise BFI items, with tailored LLMs outperforming general-purpose ones.
In this article we explore the application of Large Language Models (LLMs) in assessing the content validity of psychometric instruments, focusing on the Big Five Questionnaire (BFQ) and Big Five Inventory (BFI). Content validity, a cornerstone of test construction, ensures that psychological measures adequately cover their intended constructs. Using both human expert evaluations and advanced LLMs, we compared the accuracy of semantic item-construct alignment. Graduate psychology students employed the Content Validity Ratio (CVR) to rate test items, forming the human baseline. In parallel, state-of-the-art LLMs, including multilingual and fine-tuned models, analyzed item embeddings to predict construct mappings. The results reveal distinct strengths and limitations of human and AI approaches. Human validators excelled in aligning the behaviorally rich BFQ items, while LLMs performed better with the linguistically concise BFI items. Training strategies significantly influenced LLM performance, with models tailored for lexical relationships outperforming general-purpose LLMs. Here we highlights the complementary potential of hybrid validation systems that integrate human expertise and AI precision. The findings underscore the transformative role of LLMs in psychological assessment, paving the way for scalable, objective, and robust test development methodologies.