Transductive Off-policy Proximal Policy Optimization
This addresses a bottleneck in reinforcement learning for practitioners using PPO by enabling more efficient data utilization, though it is an incremental extension of an existing method.
The paper tackles the limitation of Proximal Policy Optimization (PPO) in using data from different policies by introducing Transductive Off-policy PPO (ToPPO), an off-policy extension with theoretical justification and guidelines, showing promising performance across six tasks.
Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due to its inherent on-policy nature, its proficiency in harnessing data from disparate policies is constrained. This paper introduces a novel off-policy extension to the original PPO method, christened Transductive Off-policy PPO (ToPPO). Herein, we provide theoretical justification for incorporating off-policy data in PPO training and prudent guidelines for its safe application. Our contribution includes a novel formulation of the policy improvement lower bound for prospective policies derived from off-policy data, accompanied by a computationally efficient mechanism to optimize this bound, underpinned by assurances of monotonic improvement. Comprehensive experimental results across six representative tasks underscore ToPPO's promising performance.