Gradient Informed Proximal Policy Optimization
This work addresses policy learning in reinforcement learning for researchers and practitioners, offering an incremental improvement by hybridizing gradient-based and PPO methods.
The paper tackles the problem of integrating analytical gradients from differentiable environments into Proximal Policy Optimization (PPO) by introducing an α-policy and adaptive metrics to manage gradient influence, resulting in outperformance over baseline algorithms in scenarios like function optimization, physics simulations, and traffic control.
We introduce a novel policy learning method that integrates analytical gradients from differentiable environments with the Proximal Policy Optimization (PPO) algorithm. To incorporate analytical gradients into the PPO framework, we introduce the concept of an α-policy that stands as a locally superior policy. By adaptively modifying the α value, we can effectively manage the influence of analytical policy gradients during learning. To this end, we suggest metrics for assessing the variance and bias of analytical gradients, reducing dependence on these gradients when high variance or bias is detected. Our proposed approach outperforms baseline algorithms in various scenarios, such as function optimization, physics simulations, and traffic control environments. Our code can be found online: https://github.com/SonSang/gippo.