Fast Adaptation via Policy-Dynamics Value Functions
This addresses the challenge for RL practitioners needing efficient adaptation to varying dynamics, though it appears incremental as it builds on existing RL methods with a novel conditioning approach.
The paper tackles the problem of slow adaptation in reinforcement learning to new environment dynamics by introducing Policy-Dynamics Value Functions (PD-VF), which enable rapid adaptation by estimating cumulative rewards in a space of policies and environments, requiring only a few actions at test time.
Standard RL algorithms assume fixed environment dynamics and require a significant amount of interaction to adapt to new environments. We introduce Policy-Dynamics Value Functions (PD-VF), a novel approach for rapidly adapting to dynamics different from those previously seen in training. PD-VF explicitly estimates the cumulative reward in a space of policies and environments. An ensemble of conventional RL policies is used to gather experience on training environments, from which embeddings of both policies and environments can be learned. Then, a value function conditioned on both embeddings is trained. At test time, a few actions are sufficient to infer the environment embedding, enabling a policy to be selected by maximizing the learned value function (which requires no additional environment interaction). We show that our method can rapidly adapt to new dynamics on a set of MuJoCo domains. Code available at https://github.com/rraileanu/policy-dynamics-value-functions.