Diversity for Contingency: Learning Diverse Behaviors for Efficient Adaptation and Transfer
This addresses the need for transferable RL agents that can adapt to changes in tasks or dynamics, though it appears incremental compared to prior diversity-focused methods.
The paper tackles the problem of discovering all possible solutions for a given task in reinforcement learning to enable efficient adaptation and transfer, proposing a method that learns diverse policies without requiring additional novelty detection models or balancing reward signals.
Discovering all useful solutions for a given task is crucial for transferable RL agents, to account for changes in the task or transition dynamics. This is not considered by classical RL algorithms that are only concerned with finding the optimal policy, given the current task and dynamics. We propose a simple method for discovering all possible solutions of a given task, to obtain an agent that performs well in the transfer setting and adapts quickly to changes in the task or transition dynamics. Our method iteratively learns a set of policies, while each subsequent policy is constrained to yield a solution that is unlikely under all previous policies. Unlike prior methods, our approach does not require learning additional models for novelty detection and avoids balancing task and novelty reward signals, by directly incorporating the constraint into the action selection and optimization steps.