Policy Prediction Network: Model-Free Behavior Policy with Model-Based Learning in Continuous Action Space
This work addresses a specific bottleneck in reinforcement learning for continuous control, offering a novel integration of model-free and model-based methods, though it is incremental in nature.
The paper tackles the challenge of applying tree-structured architectures to continuous action spaces in reinforcement learning, proposing the Policy Prediction Network to improve sample complexity and performance on continuous control problems, with experiments in MuJoCo environments showing competitive results.
This paper proposes a novel deep reinforcement learning architecture that was inspired by previous tree structured architectures which were only useable in discrete action spaces. Policy Prediction Network offers a way to improve sample complexity and performance on continuous control problems in exchange for extra computation at training time but at no cost in computation at rollout time. Our approach integrates a mix between model-free and model-based reinforcement learning. Policy Prediction Network is the first to introduce implicit model-based learning to Policy Gradient algorithms for continuous action space and is made possible via the empirically justified clipping scheme. Our experiments are focused on the MuJoCo environments so that they can be compared with similar work done in this area.