Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning
This work addresses robustness in dialogue policy learning for task-completion systems, but it is incremental as it builds on the existing Deep Dyna-Q framework.
The paper tackles the problem of Deep Dyna-Q's high dependency on simulated experience quality in dialogue policy learning by proposing Discriminative Deep Dyna-Q (D3Q), which uses an RNN-based discriminator to control training data quality, resulting in significant performance improvements over DDQ.
This paper presents a Discriminative Deep Dyna-Q (D3Q) approach to improving the effectiveness and robustness of Deep Dyna-Q (DDQ), a recently proposed framework that extends the Dyna-Q algorithm to integrate planning for task-completion dialogue policy learning. To obviate DDQ's high dependency on the quality of simulated experiences, we incorporate an RNN-based discriminator in D3Q to differentiate simulated experience from real user experience in order to control the quality of training data. Experiments show that D3Q significantly outperforms DDQ by controlling the quality of simulated experience used for planning. The effectiveness and robustness of D3Q is further demonstrated in a domain extension setting, where the agent's capability of adapting to a changing environment is tested.