What Does The User Want? Information Gain for Hierarchical Dialogue Policy Optimisation
This addresses instability in task-oriented dialogue systems for users, though it is incremental as it builds on feudal dialogue management.
The paper tackles the instability and sample inefficiency in hierarchical dialogue policy optimization by introducing an intrinsic reward based on information gain, which favors actions that resolve uncertainty or query users. Their algorithm, FeudalGain, achieves state-of-the-art results in most PyDial environments, outperforming more complex approaches, as confirmed through simulation and human trials.
The dialogue management component of a task-oriented dialogue system is typically optimised via reinforcement learning (RL). Optimisation via RL is highly susceptible to sample inefficiency and instability. The hierarchical approach called Feudal Dialogue Management takes a step towards more efficient learning by decomposing the action space. However, it still suffers from instability due to the reward only being provided at the end of the dialogue. We propose the usage of an intrinsic reward based on information gain to address this issue. Our proposed reward favours actions that resolve uncertainty or query the user whenever necessary. It enables the policy to learn how to retrieve the users' needs efficiently, which is an integral aspect in every task-oriented conversation. Our algorithm, which we call FeudalGain, achieves state-of-the-art results in most environments of the PyDial framework, outperforming much more complex approaches. We confirm the sample efficiency and stability of our algorithm through experiments in simulation and a human trial.