Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog
This addresses the problem of scaling dialog systems to complex, multi-domain tasks for developers and users, though it is incremental as it builds on existing inverse reinforcement learning methods.
The paper tackles the challenge of manually designing reward functions for multi-domain task-oriented dialog systems by proposing Guided Dialog Policy Learning, which jointly estimates rewards and optimizes policy using Adversarial Inverse Reinforcement Learning, achieving higher task success than state-of-the-art baselines.
Dialog policy decides what and how a task-oriented dialog system will respond, and plays a vital role in delivering effective conversations. Many studies apply Reinforcement Learning to learn a dialog policy with the reward function which requires elaborate design and pre-specified user goals. With the growing needs to handle complex goals across multiple domains, such manually designed reward functions are not affordable to deal with the complexity of real-world tasks. To this end, we propose Guided Dialog Policy Learning, a novel algorithm based on Adversarial Inverse Reinforcement Learning for joint reward estimation and policy optimization in multi-domain task-oriented dialog. The proposed approach estimates the reward signal and infers the user goal in the dialog sessions. The reward estimator evaluates the state-action pairs so that it can guide the dialog policy at each dialog turn. Extensive experiments on a multi-domain dialog dataset show that the dialog policy guided by the learned reward function achieves remarkably higher task success than state-of-the-art baselines.