Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search
This addresses the problem of scaling evolutionary algorithms for reinforcement learning in high-dimensional domains, representing an incremental improvement by combining existing techniques.
The paper tackled the challenge of applying Negatively Correlated Search (NCS) to large-scale reinforcement learning by developing a Cooperative Coevolution framework, resulting in outperforming three state-of-the-art deep RL methods on 10 Atari games with 50% less computational time while exploring a 1.7 million-dimensional search space.
Evolutionary algorithms (EAs) have been successfully applied to optimize the policies for Reinforcement Learning (RL) tasks due to their exploration ability. The recently proposed Negatively Correlated Search (NCS) provides a distinct parallel exploration search behavior and is expected to facilitate RL more effectively. Considering that the commonly adopted neural policies usually involves millions of parameters to be optimized, the direct application of NCS to RL may face a great challenge of the large-scale search space. To address this issue, this paper presents an NCS-friendly Cooperative Coevolution (CC) framework to scale-up NCS while largely preserving its parallel exploration search behavior. The issue of traditional CC that can deteriorate NCS is also discussed. Empirical studies on 10 popular Atari games show that the proposed method can significantly outperform three state-of-the-art deep RL methods with 50% less computational time by effectively exploring a 1.7 million-dimensional search space.