Shaped Policy Search for Evolutionary Strategies using Waypoints
This work addresses exploration inefficiencies in blackbox methods for RL, with potential applications in robotics and autonomous driving, though it appears incremental as it builds on existing ES techniques.
The paper tackles the problem of improving exploration in evolutionary strategies for reinforcement learning by using state-action pairs from trajectories to learn agent dynamics, which speeds up training, as demonstrated with experiments on Carla driving and UR5 robotic arm simulators.
In this paper, we try to improve exploration in Blackbox methods, particularly Evolution strategies (ES), when applied to Reinforcement Learning (RL) problems where intermediate waypoints/subgoals are available. Since Evolutionary strategies are highly parallelizable, instead of extracting just a scalar cumulative reward, we use the state-action pairs from the trajectories obtained during rollouts/evaluations, to learn the dynamics of the agent. The learnt dynamics are then used in the optimization procedure to speed-up training. Lastly, we show how our proposed approach is universally applicable by presenting results from experiments conducted on Carla driving and UR5 robotic arm simulators.