Dream to Chat: Model-based Reinforcement Learning on Dialogues with User Belief Modeling
This work addresses the challenge of improving dialogue systems through user belief modeling, offering a novel approach for empathetic and out-of-domain scenarios, though it is incremental in extending model-based reinforcement learning to dialogues.
The paper tackled the problem of applying world models to natural language tasks by constructing a dialogue world model that predicts user emotions, sentiments, intentions, and future utterances, resulting in state-of-the-art performances in emotion classification and sentiment identification while enhancing dialogue quality.
World models have been widely utilized in robotics, gaming, and auto-driving. However, their applications on natural language tasks are relatively limited. In this paper, we construct the dialogue world model, which could predict the user's emotion, sentiment, and intention, and future utterances. By defining a POMDP, we argue emotion, sentiment and intention can be modeled as the user belief and solved by maximizing the information bottleneck. By this user belief modeling, we apply the model-based reinforcement learning framework to the dialogue system, and propose a framework called DreamCUB. Experiments show that the pretrained dialogue world model can achieve state-of-the-art performances on emotion classification and sentiment identification, while dialogue quality is also enhanced by joint training of the policy, critic and dialogue world model. Further analysis shows that this manner holds a reasonable exploration-exploitation balance and also transfers well to out-of-domain scenarios such as empathetic dialogues.