Training Larger Networks for Deep Reinforcement Learning
This addresses a key bottleneck in scaling deep RL for broader applications, though it is incremental in improving training stability.
The paper tackled the problem of training larger neural networks in deep reinforcement learning, which previously did not improve performance due to instability, and proposed a novel method involving wider networks with DenseNet connections, decoupled representation learning, and distributed training, resulting in significant performance gains on challenging locomotion tasks.
The success of deep learning in the computer vision and natural language processing communities can be attributed to training of very deep neural networks with millions or billions of parameters which can then be trained with massive amounts of data. However, similar trend has largely eluded training of deep reinforcement learning (RL) algorithms where larger networks do not lead to performance improvement. Previous work has shown that this is mostly due to instability during training of deep RL agents when using larger networks. In this paper, we make an attempt to understand and address training of larger networks for deep RL. We first show that naively increasing network capacity does not improve performance. Then, we propose a novel method that consists of 1) wider networks with DenseNet connection, 2) decoupling representation learning from training of RL, 3) a distributed training method to mitigate overfitting problems. Using this three-fold technique, we show that we can train very large networks that result in significant performance gains. We present several ablation studies to demonstrate the efficacy of the proposed method and some intuitive understanding of the reasons for performance gain. We show that our proposed method outperforms other baseline algorithms on several challenging locomotion tasks.