Small batch deep reinforcement learning
This work addresses a critical hyperparameter tuning issue for researchers and practitioners in reinforcement learning, though it is incremental as it focuses on empirical analysis rather than a new algorithm.
The paper tackles the problem of batch size selection in deep reinforcement learning, finding that reducing batch size can lead to significant performance gains, which is surprising given the trend toward larger batches in neural network training.
In value-based deep reinforcement learning with replay memories, the batch size parameter specifies how many transitions to sample for each gradient update. Although critical to the learning process, this value is typically not adjusted when proposing new algorithms. In this work we present a broad empirical study that suggests {\em reducing} the batch size can result in a number of significant performance gains; this is surprising, as the general tendency when training neural networks is towards larger batch sizes for improved performance. We complement our experimental findings with a set of empirical analyses towards better understanding this phenomenon.