ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models
This work addresses the need for responsible integration of LLMs into public-facing applications by providing a tool to assess their values, though it is incremental as it builds on existing psychometric methods.
The authors tackled the problem of evaluating value orientations and understanding in large language models (LLMs) by introducing ValueBench, a comprehensive psychometric benchmark based on 44 inventories and 453 value dimensions, and found that LLMs can approximate expert conclusions in value-related tasks.
Large Language Models (LLMs) are transforming diverse fields and gaining increasing influence as human proxies. This development underscores the urgent need for evaluating value orientations and understanding of LLMs to ensure their responsible integration into public-facing applications. This work introduces ValueBench, the first comprehensive psychometric benchmark for evaluating value orientations and value understanding in LLMs. ValueBench collects data from 44 established psychometric inventories, encompassing 453 multifaceted value dimensions. We propose an evaluation pipeline grounded in realistic human-AI interactions to probe value orientations, along with novel tasks for evaluating value understanding in an open-ended value space. With extensive experiments conducted on six representative LLMs, we unveil their shared and distinctive value orientations and exhibit their ability to approximate expert conclusions in value-related extraction and generation tasks. ValueBench is openly accessible at https://github.com/Value4AI/ValueBench.