Goal-oriented Dialogue Policy Learning from Failures
This work addresses the challenge of inefficient dialogue policy learning for conversational AI systems, representing an incremental improvement by adapting HER to handle implicit goals in dialogues.
The paper tackles the problem of slow reinforcement learning for dialogue policies due to sparse rewards and few early successes by developing two complex hindsight experience replay (HER) methods, which outperform existing experience replay methods in learning rate as shown in experiments with a realistic user simulator.
Reinforcement learning methods have been used for learning dialogue policies. However, learning an effective dialogue policy frequently requires prohibitively many conversations. This is partly because of the sparse rewards in dialogues, and the very few successful dialogues in early learning phase. Hindsight experience replay (HER) enables learning from failures, but the vanilla HER is inapplicable to dialogue learning due to the implicit goals. In this work, we develop two complex HER methods providing different trade-offs between complexity and performance, and, for the first time, enabled HER-based dialogue policy learning. Experiments using a realistic user simulator show that our HER methods perform better than existing experience replay methods (as applied to deep Q-networks) in learning rate.