CharacterBench: Benchmarking Character Customization of Large Language Models
This work addresses the need for robust evaluation in character-based dialogue for AI researchers and developers, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of evaluating large language models' character customization capabilities by introducing CharacterBench, a bilingual generative benchmark with 22,859 human-annotated samples covering 3,956 characters across 25 categories, which outperformed state-of-the-art automatic judges like GPT-4 in experiments.
Character-based dialogue (aka role-playing) enables users to freely customize characters for interaction, which often relies on LLMs, raising the need to evaluate LLMs' character customization capability. However, existing benchmarks fail to ensure a robust evaluation as they often only involve a single character category or evaluate limited dimensions. Moreover, the sparsity of character features in responses makes feature-focused generative evaluation both ineffective and inefficient. To address these issues, we propose CharacterBench, the largest bilingual generative benchmark, with 22,859 human-annotated samples covering 3,956 characters from 25 detailed character categories. We define 11 dimensions of 6 aspects, classified as sparse and dense dimensions based on whether character features evaluated by specific dimensions manifest in each response. We enable effective and efficient evaluation by crafting tailored queries for each dimension to induce characters' responses related to specific dimensions. Further, we develop CharacterJudge model for cost-effective and stable evaluations. Experiments show its superiority over SOTA automatic judges (e.g., GPT-4) and our benchmark's potential to optimize LLMs' character customization. Our repository is at https://github.com/thu-coai/CharacterBench.