Can LLMs Assess Personality? Validating Conversational AI for Trait Profiling
This offers a promising new approach to psychometrics for personality profiling, though it is incremental as it builds on existing LLM and assessment methods.
The study validated Large Language Models (LLMs) for personality assessment by comparing Big Five scores from guided conversations with a gold-standard questionnaire, finding moderate convergent validity (r=0.38-0.58) and equivalent user-perceived accuracy.
This study validates Large Language Models (LLMs) as a dynamic alternative to questionnaire-based personality assessment. Using a within-subjects experiment (N=33), we compared Big Five personality scores derived from guided LLM conversations against the gold-standard IPIP-50 questionnaire, while also measuring user-perceived accuracy. Results indicate moderate convergent validity (r=0.38-0.58), with Conscientiousness, Openness, and Neuroticism scores statistically equivalent between methods. Agreeableness and Extraversion showed significant differences, suggesting trait-specific calibration is needed. Notably, participants rated LLM-generated profiles as equally accurate as traditional questionnaire results. These findings suggest conversational AI offers a promising new approach to traditional psychometrics.