Distributed Deep Q-Learning
This work addresses scaling reinforcement learning for game control, but it is incremental as it adapts existing frameworks to a new context.
The authors tackled the problem of learning control policies from high-dimensional sensory input by proposing a distributed deep Q-learning model, which achieved reasonable success on a simple game with minimal tuning.
We propose a distributed deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is based on the deep Q-network, a convolutional neural network trained with a variant of Q-learning. Its input is raw pixels and its output is a value function estimating future rewards from taking an action given a system state. To distribute the deep Q-network training, we adapt the DistBelief software framework to the context of efficiently training reinforcement learning agents. As a result, the method is completely asynchronous and scales well with the number of machines. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to achieve reasonable success on a simple game with minimal parameter tuning.