WALL-E: An Efficient Reinforcement Learning Research Framework
This work addresses efficiency issues in RL systems for researchers and practitioners by providing a faster framework, though it is incremental as it builds on existing multi-process architectures.
The authors tackled the bottleneck of experience collection time in reinforcement learning by developing WALL-E, a framework that uses parallel rollout samplers to accelerate rollout generation, resulting in faster convergence and higher average rewards, such as improved performance on the MuJoCo HalfCheetah-v2 task with 10 parallel processes.
There are two halves to RL systems: experience collection time and policy learning time. For a large number of samples in rollouts, experience collection time is the major bottleneck. Thus, it is necessary to speed up the rollout generation time with multi-process architecture support. Our work, dubbed WALL-E, utilizes multiple rollout samplers running in parallel to rapidly generate experience. Due to our parallel samplers, we experience not only faster convergence times, but also higher average reward thresholds. For example, on the MuJoCo HalfCheetah-v2 task, with $N = 10$ parallel sampler processes, we are able to achieve much higher average return than those from using only a single process architecture.