Reparameterized Policy Learning for Multimodal Trajectory Optimization
This work addresses a key bottleneck in reinforcement learning for robotics and control applications, offering a novel method to enhance policy learning in complex environments.
The paper tackles the challenge of parametrizing policies for reinforcement learning in high-dimensional continuous action spaces by proposing a multimodal policy framework that models policies as generative models of optimal trajectories, resulting in improved exploration and data efficiency, with empirical results showing consistent outperformance over previous approaches across various tasks.
We investigate the challenge of parametrizing policies for reinforcement learning (RL) in high-dimensional continuous action spaces. Our objective is to develop a multimodal policy that overcomes limitations inherent in the commonly-used Gaussian parameterization. To achieve this, we propose a principled framework that models the continuous RL policy as a generative model of optimal trajectories. By conditioning the policy on a latent variable, we derive a novel variational bound as the optimization objective, which promotes exploration of the environment. We then present a practical model-based RL method, called Reparameterized Policy Gradient (RPG), which leverages the multimodal policy parameterization and learned world model to achieve strong exploration capabilities and high data efficiency. Empirical results demonstrate that our method can help agents evade local optima in tasks with dense rewards and solve challenging sparse-reward environments by incorporating an object-centric intrinsic reward. Our method consistently outperforms previous approaches across a range of tasks. Code and supplementary materials are available on the project page https://haosulab.github.io/RPG/