CEM-RL: Combining evolutionary and gradient-based methods for policy search
This work addresses the problem of improving policy search methods for reinforcement learning practitioners by providing a more stable and sample-efficient approach, though it is incremental as it builds on existing combinations.
The paper tackled the trade-off between sample efficiency and stability in policy search by combining evolutionary and gradient-based methods, resulting in CEM-RL which offers a satisfactory trade-off between performance and sample efficiency on deep RL benchmarks.
Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search. The former is widely applicable and rather stable, but suffers from low sample efficiency. By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstable and highly sensitive to hyper-parameter setting. So far, these families of methods have mostly been compared as competing tools. However, an emerging approach consists in combining them so as to get the best of both worlds. Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL algorithm. In this paper, we propose a different combination scheme using the simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy gradient (td3), another off-policy deep RL algorithm which improves over ddpg. We evaluate the resulting method, cem-rl, on a set of benchmarks classically used in deep RL. We show that cem-rl benefits from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efficiency.