Diluted Near-Optimal Expert Demonstrations for Guiding Dialogue Stochastic Policy Optimisation
This work is significant for dialogue system developers, as it provides methods to accelerate the learning process of dialogue agents, especially in data-scarce environments.
This paper addresses the scarcity of human interaction data for learning dialogue agents by proposing imitation learning strategies that use a near-optimal handcrafted policy as an expert. Their hybridisation strategy, which seamlessly combines the learned policy with the expert, outperforms several baselines and accelerates learning when interacting with real humans.
A learning dialogue agent can infer its behaviour from interactions with the users. These interactions can be taken from either human-to-human or human-machine conversations. However, human interactions are scarce and costly, making learning from few interactions essential. One solution to speedup the learning process is to guide the agent's exploration with the help of an expert. We present in this paper several imitation learning strategies for dialogue policy where the guiding expert is a near-optimal handcrafted policy. We incorporate these strategies with state-of-the-art reinforcement learning methods based on Q-learning and actor-critic. We notably propose a randomised exploration policy which allows for a seamless hybridisation of the learned policy and the expert. Our experiments show that our hybridisation strategy outperforms several baselines, and that it can accelerate the learning when facing real humans.