Large Language Models Meet Harry Potter: A Bilingual Dataset for Aligning Dialogue Agents with Characters
This work addresses the problem of character alignment for dialogue agents, providing a dataset and benchmark for researchers, but it is incremental as it builds on existing LLM capabilities without a major breakthrough.
The authors tackled the challenge of aligning dialogue agents with specific characters by introducing the bilingual Harry Potter Dialogue dataset, which includes extensive annotations and serves as a benchmark, revealing that models show potential but need improvement in generating character-aligned responses.
In recent years, Dialogue-style Large Language Models (LLMs) such as ChatGPT and GPT4 have demonstrated immense potential in constructing open-domain dialogue agents. However, aligning these agents with specific characters or individuals remains a considerable challenge due to the complexities of character representation and the lack of comprehensive annotations. In this paper, we introduce the Harry Potter Dialogue (HPD) dataset, designed to advance the study of dialogue agents and character alignment. The dataset encompasses all dialogue sessions (in both English and Chinese) from the Harry Potter series and is annotated with vital background information, including dialogue scenes, speakers, character relationships, and attributes. These extensive annotations may empower LLMs to unlock character-driven dialogue capabilities. Furthermore, it can serve as a universal benchmark for evaluating how well can a LLM aligning with a specific character. We benchmark LLMs on HPD using both fine-tuning and in-context learning settings. Evaluation results reveal that although there is substantial room for improvement in generating high-quality, character-aligned responses, the proposed dataset is valuable in guiding models toward responses that better align with the character of Harry Potter.