Multitask Neuroevolution for Reinforcement Learning with Long and Short Episodes
This addresses the problem of sample inefficiency in evolutionary reinforcement learning for continuous control tasks, though it appears incremental as it builds on existing ES methods with a novel transfer mechanism.
The paper tackles the high sample complexity of evolution strategies in deep reinforcement learning over long horizons by introducing a neuroevolutionary multitasking algorithm that transfers skills from auxiliary short-episode tasks to the target long-episode task, achieving data-efficient RL with reduced agent-environment interaction requirements.
Studies have shown evolution strategies (ES) to be a promising approach for reinforcement learning (RL) with deep neural networks. However, the issue of high sample complexity persists in applications of ES to deep RL over long horizons. This paper is the first to address the shortcoming of today's methods via a novel neuroevolutionary multitasking (NuEMT) algorithm, designed to transfer information from a set of auxiliary tasks (of short episode length) to the target (full length) RL task at hand. The auxiliary tasks, extracted from the target, allow an agent to update and quickly evaluate policies on shorter time horizons. The evolved skills are then transferred to guide the longer and harder task towards an optimal policy. We demonstrate that the NuEMT algorithm achieves data-efficient evolutionary RL, reducing expensive agent-environment interaction data requirements. Our key algorithmic contribution in this setting is to introduce, for the first time, a multitask skills transfer mechanism based on the statistical importance sampling technique. In addition, an adaptive resource allocation strategy is utilized to assign computational resources to auxiliary tasks based on their gleaned usefulness. Experiments on a range of continuous control tasks from the OpenAI Gym confirm that our proposed algorithm is efficient compared to recent ES baselines.