Structured Hierarchical Dialogue Policy with Graph Neural Networks
This addresses the challenge of training dialogue policies for composite tasks like restaurant reservations, offering incremental improvements in efficiency and transferability for practical dialogue systems.
The paper tackled the problem of low sampling efficiency and poor transferability in hierarchical deep reinforcement learning for composite dialogue tasks by proposing a novel ComNet using graph neural networks. The result showed that ComNet outperformed vanilla HDRL systems, achieving performance close to the upper bound with improved sample efficiency, robustness to noise, and transferability.
Dialogue policy training for composite tasks, such as restaurant reservation in multiple places, is a practically important and challenging problem. Recently, hierarchical deep reinforcement learning (HDRL) methods have achieved good performance in composite tasks. However, in vanilla HDRL, both top-level and low-level policies are all represented by multi-layer perceptrons (MLPs) which take the concatenation of all observations from the environment as the input for predicting actions. Thus, traditional HDRL approach often suffers from low sampling efficiency and poor transferability. In this paper, we address these problems by utilizing the flexibility of graph neural networks (GNNs). A novel ComNet is proposed to model the structure of a hierarchical agent. The performance of ComNet is tested on composited tasks of the PyDial benchmark. Experiments show that ComNet outperforms vanilla HDRL systems with performance close to the upper bound. It not only achieves sample efficiency but also is more robust to noise while maintaining the transferability to other composite tasks.