AIPsychoBench: Understanding the Psychometric Differences between LLMs and Humans
This addresses the need for reliable and interpretable psychometric evaluation of LLMs, particularly for researchers and developers, though it is incremental as it builds on existing assessment methods.
The paper tackles the problem of assessing psychological properties in LLMs by introducing AIPsychoBench, a specialized benchmark that improves the average effective response rate from 70.12% to 90.40% and reduces biases compared to traditional methods, while also providing evidence of linguistic impacts with score deviations ranging from 5% to 20.2% across 43 subcategories.
Large Language Models (LLMs) with hundreds of billions of parameters have exhibited human-like intelligence by learning from vast amounts of internet-scale data. However, the uninterpretability of large-scale neural networks raises concerns about the reliability of LLM. Studies have attempted to assess the psychometric properties of LLMs by borrowing concepts from human psychology to enhance their interpretability, but they fail to account for the fundamental differences between LLMs and humans. This results in high rejection rates when human scales are reused directly. Furthermore, these scales do not support the measurement of LLM psychological property variations in different languages. This paper introduces AIPsychoBench, a specialized benchmark tailored to assess the psychological properties of LLM. It uses a lightweight role-playing prompt to bypass LLM alignment, improving the average effective response rate from 70.12% to 90.40%. Meanwhile, the average biases are only 3.3% (positive) and 2.1% (negative), which are significantly lower than the biases of 9.8% and 6.9%, respectively, caused by traditional jailbreak prompts. Furthermore, among the total of 112 psychometric subcategories, the score deviations for seven languages compared to English ranged from 5% to 20.2% in 43 subcategories, providing the first comprehensive evidence of the linguistic impact on the psychometrics of LLM.