LLM Roleplay: Simulating Human-Chatbot Interaction
This addresses the need for scalable dialogue generation in chatbot development, though it is incremental as it builds on existing LLM capabilities.
The paper tackles the problem of high resource requirements for collecting diverse human-chatbot dialogues by proposing LLM Roleplay, a method that automatically generates such dialogues using large language models to simulate personas, achieving a high indistinguishability rate compared to natural dialogues.
The development of chatbots requires collecting a large number of human-chatbot dialogues to reflect the breadth of users' sociodemographic backgrounds and conversational goals. However, the resource requirements to conduct the respective user studies can be prohibitively high and often only allow for a narrow analysis of specific dialogue goals and participant demographics. In this paper, we propose LLM Roleplay: a goal-oriented, persona-based method to automatically generate diverse multi-turn dialogues simulating human-chatbot interaction. LLM Roleplay can be applied to generate dialogues with any type of chatbot and uses large language models (LLMs) to play the role of textually described personas. To validate our method, we collect natural human-chatbot dialogues from different sociodemographic groups and conduct a user study to compare these with our generated dialogues. We evaluate the capabilities of state-of-the-art LLMs in maintaining a conversation during their embodiment of a specific persona and find that our method can simulate human-chatbot dialogues with a high indistinguishability rate.