Model Based Meta Learning of Critics for Policy Gradients
This addresses the challenge of generalization in robotics and reinforcement learning, though it appears incremental as it builds on existing meta-learning and policy gradient methods.
The paper tackles the problem of learning representations that generalize quickly to new scenarios in reinforcement learning by meta-learning a critic for policy gradients, resulting in a critic that can learn new policies for unseen tasks without requiring a model.
Being able to seamlessly generalize across different tasks is fundamental for robots to act in our world. However, learning representations that generalize quickly to new scenarios is still an open research problem in reinforcement learning. In this paper we present a framework to meta-learn the critic for gradient-based policy learning. Concretely, we propose a model-based bi-level optimization algorithm that updates the critics parameters such that the policy that is learned with the updated critic gets closer to solving the meta-training tasks. We illustrate that our algorithm leads to learned critics that resemble the ground truth Q function for a given task. Finally, after meta-training, the learned critic can be used to learn new policies for new unseen task and environment settings via model-free policy gradient optimization, without requiring a model. We present results that show the generalization capabilities of our learned critic to new tasks and dynamics when used to learn a new policy in a new scenario.