Evolution Strategies for Deep RL pretraining
This work addresses the computational cost and parameter sensitivity of DRL for researchers, but it is incremental as it shows limited practical benefits for ES pretraining.
This study tackled the problem of using evolution strategies (ES) for deep reinforcement learning (DRL) pretraining to improve efficiency, finding that ES did not consistently train faster than DRL and only provided benefits in less complex environments like Flappy Bird, with minimal or no improvement in more sophisticated tasks such as Breakout and MuJoCo Walker.
Although Deep Reinforcement Learning has proven highly effective for complex decision-making problems, it demands significant computational resources and careful parameter adjustment in order to develop successful strategies. Evolution strategies offer a more straightforward, derivative-free approach that is less computationally costly and simpler to deploy. However, ES generally do not match the performance levels achieved by DRL, which calls into question their suitability for more demanding scenarios. This study examines the performance of ES and DRL across tasks of varying difficulty, including Flappy Bird, Breakout and Mujoco environments, as well as whether ES could be used for initial training to enhance DRL algorithms. The results indicate that ES do not consistently train faster than DRL. When used as a preliminary training step, they only provide benefits in less complex environments (Flappy Bird) and show minimal or no improvement in training efficiency or stability across different parameter settings when applied to more sophisticated tasks (Breakout and MuJoCo Walker).