A Globally Convergent Evolutionary Strategy for Stochastic Constrained Optimization with Applications to Reinforcement Learning
This work addresses the problem of ensuring convergence in evolutionary strategies for stochastic constrained optimization, which is important for researchers and practitioners in reinforcement learning, though it appears incremental as it builds on existing evolutionary strategy methods.
The authors tackled the lack of convergence guarantees for evolutionary strategies in stochastic constrained optimization problems, such as those in reinforcement learning, by designing a novel algorithm with a sufficient decrease mechanism that ensures convergence based on function estimates, and demonstrated its applicability on control tasks and constrained reward maximization.
Evolutionary strategies have recently been shown to achieve competing levels of performance for complex optimization problems in reinforcement learning. In such problems, one often needs to optimize an objective function subject to a set of constraints, including for instance constraints on the entropy of a policy or to restrict the possible set of actions or states accessible to an agent. Convergence guarantees for evolutionary strategies to optimize stochastic constrained problems are however lacking in the literature. In this work, we address this problem by designing a novel optimization algorithm with a sufficient decrease mechanism that ensures convergence and that is based only on estimates of the functions. We demonstrate the applicability of this algorithm on two types of experiments: i) a control task for maximizing rewards and ii) maximizing rewards subject to a non-relaxable set of constraints.