PRODIGy: a PROfile-based DIalogue Generation dataset
This work addresses the need for better profile-based dialogue datasets to enhance conversational AI, though it is incremental as it builds on existing methods with a new resource.
The authors tackled the problem of improving dialogue agent consistency and coherence by introducing PRODIGy, a dataset that aligns dialogues with multiple speaker profile representations, and found that profile-based models outperformed dialogue-only models in both automatic and human evaluations.
Providing dialogue agents with a profile representation can improve their consistency and coherence, leading to better conversations. However, current profile-based dialogue datasets for training such agents contain either explicit profile representations that are simple and dialogue-specific, or implicit representations that are difficult to collect. In this work, we propose a unified framework in which we bring together both standard and more sophisticated profile representations by creating a new resource where each dialogue is aligned with all possible speaker representations such as communication style, biographies, and personality. This framework allows to test several baselines built using generative language models with several profile configurations. The automatic evaluation shows that profile-based models have better generalisation capabilities than models trained on dialogues only, both in-domain and cross-domain settings. These results are consistent for fine-tuned models and instruction-based LLMs. Additionally, human evaluation demonstrates a clear preference for generations consistent with both profile and context. Finally, to account for possible privacy concerns, all experiments are done under two configurations: inter-character and intra-character. In the former, the LM stores the information about the character in its internal representation, while in the latter, the LM does not retain any personal information but uses it only at inference time.