Effects of Different Optimization Formulations in Evolutionary Reinforcement Learning on Diverse Behavior Generation
This work addresses the challenge of generating varied strategies in reinforcement learning, which is incremental as it analyzes existing frameworks to improve behavior exploration.
The paper investigates how different optimization formulations in evolutionary reinforcement learning affect the generation of diverse behaviors, finding that formulations that do not treat objectives equally fail to produce diversity and result in worse-performing agents in Atari games.
Generating various strategies for a given task is challenging. However, it has already proven to bring many assets to the main learning process, such as improved behavior exploration. With the growth in the interest of heterogeneity in solution in evolutionary computation and reinforcement learning, many promising approaches have emerged. To better understand how one guides multiple policies toward distinct strategies and benefit from diversity, we need to analyze further the influence of the reward signal modulation and other evolutionary mechanisms on the obtained behaviors. To that effect, this paper considers an existing evolutionary reinforcement learning framework which exploits multi-objective optimization as a way to obtain policies that succeed at behavior-related tasks as well as completing the main goal. Experiments on the Atari games stress that optimization formulations which do not consider objectives equally fail at generating diversity and even output agents that are worse at solving the problem at hand, regardless of the obtained behaviors.