BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems
This addresses the challenge of slow exploration in dialogue systems, which is incremental as it builds on existing deep reinforcement learning methods.
The paper tackles the problem of inefficient exploration in deep Q-learning for task-oriented dialogue systems by introducing an algorithm that uses Thompson sampling with Bayes-by-Backprop neural networks, resulting in significantly faster learning compared to common exploration strategies.
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as ε-greedy, Boltzmann, bootstrapping, and intrinsic-reward-based ones. Additionally, we show that spiking the replay buffer with experiences from just a few successful episodes can make Q-learning feasible when it might otherwise fail.