IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks
This addresses the training time bottleneck for practitioners using scalable reinforcement learning, though it is incremental as it builds on IMPALA.
The paper tackled the trade-off between sample throughput and sample efficiency in distributed reinforcement learning by proposing IMPACT, a new algorithm that extends IMPALA with target networks, a circular buffer, and truncated importance sampling, achieving up to 30% faster training in discrete environments while maintaining sample efficiency in continuous control.
The practical usage of reinforcement learning agents is often bottlenecked by the duration of training time. To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. However, modern methods for scalable reinforcement learning (RL) often tradeoff between the throughput of samples that an RL agent can learn from (sample throughput) and the quality of learning from each sample (sample efficiency). In these scalable RL architectures, as one increases sample throughput (i.e. increasing parallelization in IMPALA), sample efficiency drops significantly. To address this, we propose a new distributed reinforcement learning algorithm, IMPACT. IMPACT extends IMPALA with three changes: a target network for stabilizing the surrogate objective, a circular buffer, and truncated importance sampling. In discrete action-space environments, we show that IMPACT attains higher reward and, simultaneously, achieves up to 30% decrease in training wall-time than that of IMPALA. For continuous control environments, IMPACT trains faster than existing scalable agents while preserving the sample efficiency of synchronous PPO.