PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replay
This addresses a bottleneck in reinforcement learning for researchers and practitioners by improving sampling efficiency, though it is incremental as it builds on existing PPO methods.
The paper tackles the low data efficiency of on-policy deep reinforcement learning by proposing PTR-PPO, which combines on-policy and off-policy methods with prioritized trajectory replay, achieving state-of-the-art performance on Atari discrete control tasks.
On-policy deep reinforcement learning algorithms have low data utilization and require significant experience for policy improvement. This paper proposes a proximal policy optimization algorithm with prioritized trajectory replay (PTR-PPO) that combines on-policy and off-policy methods to improve sampling efficiency by prioritizing the replay of trajectories generated by old policies. We first design three trajectory priorities based on the characteristics of trajectories: the first two being max and mean trajectory priorities based on one-step empirical generalized advantage estimation (GAE) values and the last being reward trajectory priorities based on normalized undiscounted cumulative reward. Then, we incorporate the prioritized trajectory replay into the PPO algorithm, propose a truncated importance weight method to overcome the high variance caused by large importance weights under multistep experience, and design a policy improvement loss function for PPO under off-policy conditions. We evaluate the performance of PTR-PPO in a set of Atari discrete control tasks, achieving state-of-the-art performance. In addition, by analyzing the heatmap of priority changes at various locations in the priority memory during training, we find that memory size and rollout length can have a significant impact on the distribution of trajectory priorities and, hence, on the performance of the algorithm.