High-Throughput Synchronous Deep RL
This work provides a more efficient and stable deep reinforcement learning training method for researchers and practitioners, addressing the trade-off between throughput and stability in existing approaches.
This paper addresses the computational demands of deep reinforcement learning by proposing High-Throughput Synchronous Deep Reinforcement Learning (HTS-RL). HTS-RL combines the stability of synchronous methods with high throughput by concurrently performing learning and rollouts, avoiding stale policies, and ensuring asynchronous actor-environment interaction while maintaining determinism. HTS-RL is 2-6x faster than synchronous baselines and achieves higher average episode rewards than state-of-the-art asynchronous methods with competitive throughput.
Deep reinforcement learning (RL) is computationally demanding and requires processing of many data points. Synchronous methods enjoy training stability while having lower data throughput. In contrast, asynchronous methods achieve high throughput but suffer from stability issues and lower sample efficiency due to `stale policies.' To combine the advantages of both methods we propose High-Throughput Synchronous Deep Reinforcement Learning (HTS-RL). In HTS-RL, we perform learning and rollouts concurrently, devise a system design which avoids `stale policies' and ensure that actors interact with environment replicas in an asynchronous manner while maintaining full determinism. We evaluate our approach on Atari games and the Google Research Football environment. Compared to synchronous baselines, HTS-RL is 2-6$\times$ faster. Compared to state-of-the-art asynchronous methods, HTS-RL has competitive throughput and consistently achieves higher average episode rewards.