Novel Policy Seeking with Constrained Optimization
This work addresses the lack of diversity in reinforcement learning policies, which is an incremental advance for AI systems requiring varied problem-solving strategies.
The paper tackles the problem of generating novel policies in reinforcement learning, which typically produce monotonous solutions, by proposing a constrained optimization approach that introduces two methods (CTNB and IPD) to improve policy novelty and task performance, with experimental results on MuJoCo showing substantial improvements over previous methods.
In problem-solving, we humans can come up with multiple novel solutions to the same problem. However, reinforcement learning algorithms can only produce a set of monotonous policies that maximize the cumulative reward but lack diversity and novelty. In this work, we address the problem of generating novel policies in reinforcement learning tasks. Instead of following the multi-objective framework used in existing methods, we propose to rethink the problem under a novel perspective of constrained optimization. We first introduce a new metric to evaluate the difference between policies and then design two practical novel policy generation methods following the new perspective. The two proposed methods, namely the Constrained Task Novel Bisector (CTNB) and the Interior Policy Differentiation (IPD), are derived from the feasible direction method and the interior point method commonly known in the constrained optimization literature. Experimental comparisons on the MuJoCo control suite show our methods can achieve substantial improvement over previous novelty-seeking methods in terms of both the novelty of policies and their performances in the primal task.